A factory manager in an RMG buying house is responsible for overseeing the production of garments, ensuring they meet the buying house's specifications and quality standards. They coordinate various modules of production, manage staff, and liaise with different departments to ensure timely and efficient production.