Some understanding of this problem is necessary for anyone interested in security and privacy, so I offer two good books that will provide enough background for most of us. Naturally each has pointers for further reading. Both books are written for mathematicians not computer security professionals, although both have sections on public keys and Reisel has Pascal code for various factorization methods. Having been available for a couple of years Reisel's book has already been deemed worthwhile. There are only seven chapters, but there are nine appendices.
Factoring is broken into two chapters, one on classical method and one on modern methods.
The tables are composed of primes, factors in many formats, quadratic residues and formulas for cyclotomic polynomials. There is even an appendix devoted to elliptic curves. He begins with a good introduction the concept of a prime number and the prime number theorem. Alongside this he describes the basic sieve of Eratosthenes in works, mathematical terms and Pascal code. This helpful approach is maintained throughout the book.