If your child is quite young (say 6 months to 2 years old) when you go to prison, then yes it can. It will be hard for them to remember who you are. They are likely to form a bond with the person looking after them. Similarly if you are in prison for a long time your child is likely to become attached to the caregiver and the community they are with. If this happens, the court may not think that it’s in the best interests of the child to give you day to day care.
The best thing you can do is make the effort to keep in contact with your child as often as you can while you are in prison. This will allow the child to have a bond with you as well.
If your child has lost touch with you, you could begin a gradual process of getting familiar again. If you don’t have day to day care when you leave prison, the court can give you contact. This will give your child time to get used to you and to re-establish a parent-child relationship.
However, there are no guarantees that you will get your child back, even if you do these things. It will definitely improve your chances but it’s ultimately up to the court and what they believe is in your child’s best interests.