Allocating a set object and hashing each byte to perform this computation is way more expensive than it would be to just iterate over the last `pad_size` bytes and check their values.