I hope that I haven't done anything to damage our trust. This is a difficult question to answer in a few paragraphs. Parrots can be highly empathic, responding to every nuance of their owner's mood and energy. It may very well be that your lory did sense that you were uncomfortable with him and therefore, was not comfortable with you. It could also be that your Rainbow lory was not really that tame.
Lories are generally excitable and sometimes fairly aggressive little birds.
They need a lot of handling and guidance when they are young to gentle them and keep them tame. If your lory was handfed but then weaned and not handled that much out of his cage until you brought him home, he might have become quite territorial about his cage because that was what he was used to and where he feels safe. Your statement that the fact that your lory does not bathe or play with is toys is normal for lorikeets couldn't be further from the truth. Lories are one of the most playful acrobatic birds available as pets and normally love to bathe. Anything that you do that threatens your bird will threaten the trust you are trying to build with him, and that will damage his bond to you.