Memory priority is not going to guarantee any limitation on RAM use, and there's no good way to achieve that objective. You can create a Watchdog rule to trim the working set of a process when it exceeds a certain value, but that memory will be paged right back in if the process is actively using it, and if the process wasn't using that memory, then it'd be naturally paged out over time anyway.
MSDN says:
"The memory priority of a thread or process serves as a hint to the memory manager when it trims pages from the working set. Other factors being equal, pages with lower memory priority are trimmed before pages with higher memory priority. For more information, see Working Set."