libroot: explicitly check ABI version

As Axel suggested use simple, explicit checks for legacy ABI version
instead of obscure "compatibility mode".
vm: fix area insertion logic

 * When looking for a place for new area the size of the area to be
   inserted instead of the next area size was used to check whether
   we are already past the upper bound.
 * There was an attempt to insert area even if we were past the
   upper bound.