if the folder doesn't exist, the agent complains that the folder does not exist:
error: Directory '/opt/rudder/etc/server-roles.d/' could not be accessed in lsdir(), (opendir: No such file or directory)

If folder is empty, it does not complain

In both cases, the inventory is valid, with empty <SERVER_ROLES> tag