He will never learn to trust you. That is very hard. But you don't have that problem. The key to success is persistance. The bird is hoping you will give up and leave him alone.
Most people give up.
Your problem is not too bad since you said he doesn't try to bite you. Trying going into a quiet room just the two of you where there is not alot of noise. Pets and children running around can scare a bird. If the bird's cage is located where there is alot of traffic this may play a part in the bird feeling uneasy. Too much going on all the time.