Change joining against idsetmember table to use a cross join with where clause (which results in an inner join) rather than as an in clause.  Mysql 5.5 does not perform sufficiently well with the in clause, and most implementations still use that.