Wierdness in RAM accounting

Since Ubuntu 14 /Deb 7guests I experienced a wierdness in OpenVZ's
RAM accounting. Without vSwap it is completely false, seems like the
kernel would not want to drop the caches/buffers when needed, instead
memory allocation fails.
After a lot of search I found only traces that others experience it, too,
but no solution.
After setting vSwap, I set privvmpages to unlimited, or it will still limit
the ammount of memory reserved (NOT used, solely reserved) by the client.