Most of the valuable games are ones that came out at the end of the system's life when fewer people owned the system. There are exceptions to this rule. The better the game is, the less chance it has much value. Most of the valuable games are the ones that were so bad no one wanted them. Again, there are exceptions to this rule.
Rare cartridges vary in value from system to system.
A rare game on the Atari 2600 is generally more valuable than one on the Intellivision, mainly due to the increased amount of collectors. Rulebooks and boxes are usually tougher to find than the actual carts. While they are usually not worth as much as the cart, that is quickly changing. Since they have no moving parts, they are quite tough to destroy. Out of over 3,500 games I have owned, I only came across three that wouldn't work.