Allah Erased Memory Of Ahd-e-Alast Doesn't Make Sense
Adam’s (pbuh) memory doesn’t seem to be erased because he was in heaven and he remembered that eating the fruit was prohibited.
Yes, limitations of our brain could be the reason why we don’t remember prebirth life. Since our life on earth is subject to the natural laws and therefore it would be too much to expect to remember anything from the prebirth period. It would need a miracle.