If you pay by check, the tour operator's brochure usually will specify the name of an escrow bank account where all payments eventually go. Make out your check to that account. Also, if possible, put the destination, dates, and other details on the face of the check, which should guarantee that the payment goes where it should go. That may help you get your money back if the tour is canceled or if the tour operator or travel agent goes out of business. Demand it if it isn't offered to you.
Some operators carry bonds to reassure their customers, rather than using escrow accounts.
If an operator doesn't have an escrow account, ask whether it is bonded and how you would be reimbursed in case of a default. Often, the travel agent will insist that the check be made out to the travel agency, because it is the policy of some agents to write a single check to the tour operator themselves. That's fine, but insist on a written guarantee from the tour operator and the agency, and make sure that the agency's check is made payable to the tour operator's escrow account. Reputable agents and operators should be willing to stand behind the tour. You also can protect yourself by getting trip insurance.