Glycogen does not make you fat. The only thing that can increase body fat is consuming more calories than you burn while not using them to build muscle. Consuming more calories than you burn is also necessary for building muscle mass.