Revert "Invenotory: Change to use inventory cache even if we do not specify the date."

Needs more tests.

This reverts commit 74db4d392ff47c248bb070b9cda4a65560b989c2.