Revert "Remove special cases from the CI matrix"

This reverts commit f3a8a3469e698085e458fec525e676593c24ed9f.

See https://github.com/dkubb/memoizable/commit/f3a8a3469e698085e458fec525e676593c24ed9f#commitcomment-4630045