Support restricting grant types by client

if grant_types are not defined, then none are restricted