Ok, so it also does not make sense to move

$this->_serializer->setGroups($this->_serializerGroups);

to setSerializer as otherwise it could be affected by external changes.