got it! better hand tracking actually made it possible to use a very simple matching solution between frames. works pretty well on live data now.