Lock down what can use the callback filter

Just worried this might become a security risk down the road. Decided to lock it down before then.