It may be a factor as far as child support obligations go (for example, if you lived there and are making contributions to the household), then that would be a factor.  But as for custody, unless you pose a danger or threat to the child, then there should not be a problem.