The theoretical results of the paper can be extended to accommodate non-differentiable loss functions by employing subgradient methods or generalized gradients. In scenarios where the loss function is not differentiable, one can utilize the concept of a subgradient, which generalizes the notion of a gradient to non-differentiable functions. This allows for the definition of a "subgradient descent" approach, where updates can still be framed in the context of natural gradient descent by substituting the traditional gradient with a subgradient.
Moreover, the framework can be adapted to handle more general performance measures beyond scalar functions by considering vector-valued performance metrics. This can be achieved by defining a multi-dimensional loss landscape and employing techniques such as vector optimization or Pareto optimization, where the goal is to minimize a vector of losses simultaneously. By reformulating the learning rules to account for these broader performance measures, one can still express updates in the natural gradient form, thereby maintaining the theoretical underpinnings established in the paper.