He does not bath or play with his toys, which I understand are typical for lorikeets. Out of his cage, he is willing to be hand fed and loves to be scratched on his head. He lets himself be handled by me and others. I hope that I haven't done anything to damage our trust. This is a difficult question to answer in a few paragraphs.
Parrots can be highly empathic, responding to every nuance of their owner's mood and energy.
It may very well be that your lory did sense that you were uncomfortable with him and therefore, was not comfortable with you. It could also be that your Rainbow lory was not really that tame. Lories are generally excitable and sometimes fairly aggressive little birds. They need a lot of handling and guidance when they are young to gentle them and keep them tame. If your lory was handfed but then weaned and not handled that much out of his cage until you brought him home, he might have become quite territorial about his cage because that was what he was used to and where he feels safe.