There is no chance for you to lose your followers by making your Instagram account private. Additionally, they will not get affected in any way by this action of yours.