Limiting memory use is difficult and requires Watchdog rules to restart or terminate offending processes when they exceed a threshold. Forcibly denying them allocation requests would cause them to simply crash. In some cases, slowing them down by constraining their CPU affinity (e.g. CPU limiter) can also reduce the rate of memory consumption.
Of course, users can ‘trim’ the virtual memory of a process, but that isn’t going to be effective if the virtual memory is actively referenced since it will just get paged right back in as soon as it is referenced. This is usually not a good idea since Windows will page out seldom referenced virtual memory anyway.