estimate geometric transformation