Vision is GUI and Verilog Code from BinPy, that's why connector is there. 
To what you are saying is not wrong, but think of a circuit in which gate1 and gate2 are connected with a wire, then same wire can be removed from these gates and can be put intp some gate3 and gate4 , then my circuit should work , and it will work with this implementation..
But i understood your doubt , the problem is not with changing the input owner or output owner, problem lies that when we are changing the gates, we are not removing it from the earlier ones. I think we should do that rather than restricting that connector to be used as connection between other gates.