Whether or not you have started yet is not the issue; the only issue is what the offer letter says. An offer letter can create an enforceable agreement or contract; if it does, then either party may enforce its terms against the other. So if the letter essentially indicates that you *will* take the job and start either on a given date or within a reasonable time, then your prospective employer could technically bring a legal action to enforce it--not so much to force you to work for them, but to recover costs expended as a result of recruiting you and *possibly* (depending on the exact terms) to keep you from working for a competitor. To understand what, if anything, you are  obligated to, you need to review the terms of the document you signed; an employment lawyer can help you review it and understand it.