Fixes #1183 PBC bug in g_mindist

Fixes minimum size of the box if triclinic when checking
the periodic image distance.

Change-Id: I54cb593c42f791b6540147233c345069f84e2f33