Earlier works of an author tend to be more desirable. Avoid modern paperbacks, Book Club editions, former library books, textbooks and outdated reference books, older books on popular topics such as health care and business, condensed versions of popular books. If you wish to collect such books anyway, you should pay very little for them. Set limits without limiting yourself. True, it's an oxymoron, but try to gain focus without losing imagination.
Condition is the most important factor in the value of a book.
They will pay for themselves many times over. Read reference books, dealers' catalogues, and anything about books you can get your hands on. Bookshops surface in the most unlikely of places, so don't get caught without your want list or any other guides you may need. I never met a collector who regretted making a particular purchase. I have met hundreds who regretted not making one.