It is fine to change the permissions of the dir but to recurse cand open reading to all can expose sensitive information. Depending on the environment the keys may be sensitive, it also appears that at least in RHEL 7 it is possible to set a password to decrypt private keys, that file is by default located in /etc/ntp/crypto/pw, which with recursion turned on is changed to mode 0755.