To do this requires forcibly shutting down applications in that user session when the memory threshold is exceeded.
It is possible using Watchdog rules, but these would be specific to each process (rather than cumulative for the entire session). You could create a rule, for instance, that would terminate any process in a user session ‘Bob’ when it exceeded a virtual memory limit.
The match string might look like:
winword.exe:bob
Wildcards, or even Regular Expressions, can be used like:
win*.exe:bo*
However, you should be as selective as possible when creating match strings.
The resulting Watchdog entry might look like below.
Watchdog Memory Limit Rule for instances of Chrome.exe and user ‘Bob’