The answer to the riddle is Post Office. Post Office starts with P and ends with E. Post Office has thousands of letters.