It might be unrealistic to assume that the receptive fields of auditory neurons have so much diversity as to include all possible HRTFs.
It is not straightforward to see how the auditory system could learn these acoustical filters.