You can't turn off Memories completely, but you can use the Hide Dates feature (Memories > Hide Dates > Add Date Range) to do a workaround. Enter a date range that covers your entire presence on Facebook, and Memories won't show up.