Infinite Scroll: Do not enqueue all scripts on single pages

Fixes #634
Previous commit 19fd934d9d29684bd0535851fd5b2215c40af66c didn't completely fix things