How they could know this is not always obvious. In a benign environment, it could be as simple as looking at your application and knowing or reverse-engineering the algorithm the application employs. In a more hostile environment, the enemy has to apply other intelligence-gathering techniques to find out how you encrypt your messages. The book assumes rightly that such techniques exist and are effective. If the snoop learns the key as well as the algorithm, then no codebreaking is necessary.
Many encryption algorithms depend on the use of a random-number generator.
If my computer and your computer both generate identical random-number sequences given the same seed, then we can build a reasonably secure cryptosystem, or so it would seem. As long as the sequence is truly random with no repeating patterns, there are no patterns for the codebreaker to observe. The book points out the problem with these schemes. Give them enough cycles, and they will repeat themselves. Repetitions produce patterns, on which a cryptanalyst thrives.