Looks like all toilet blowers use a common pipe without non-return dampers.
So when other apartment blower was ON, odors from other toilets were entering the room through WC blower hole which was OFF. As a consequence, I had to keep toilet fan ON all night long to avoid this, having the fan noise disturbing my sleep.
Even though, it was hard to keep the odors away!!