Your question touches on a principle known as adverse possession.  There are various elements involved in proving adverse possession, one of which is that the possession must be adverse.  New York revised its rule on adverse possession a few years back to prevent de minimis fence encroachments from suppporting a claim for adverse possession - the statute states that de minimis fence encroachments are non adverse -  meaning that you cannot establish the element of hostility and therefore would not be able to establish the fence line as the boundary line.  The statute does not define de minimis and the cases that attempt to interprete vary. I have seen a case that would not allow a claim for adverse possession where the fence encroached 10 feet.