The Incredible Machine is a series of puzzles, which includes a toy-like construction set for building puzzles. SimCity is a toy, which players make more puzzle-like by setting their own goals. Myst is a story, which happens to be told partly through puzzles. The player should have fun just manipulating the puzzle, even before reaching a solution. For instance, players can enjoy rotating and manipulating blocks in the action puzzle game Tetris even if they don't understand the goal.
Puzzles are solitary affairs, unlike games.
One of the main challenges for me as a game designer is to find ways to introduce some of the thrill of competition back into puzzle games. As Maroney points out, the card game Solitaire is an interesting borderline case between game and puzzle. Shuffling the cards is a way to randomly generate a new puzzle. Any given puzzle may require more of one or the other. For instance, my letter puzzle emphasizes ingenuity, while mazes emphasize patience.