No, the contractor cannot do this. If you are a subcontactor and performed work/provided services pursuant to some agreement (oral or written), and you honored all your obligations, you must be paid for the work. The contractor could only withhold payment to you if there was an agreement in place between you allowing him to withhold or deduct what you owe him from what he owes you--if there is such an agreement, it could be enforced against you, but otherwise he has to pay you.As a practical matter, rather than suing for the money, if you admit you owe him a smaller amount, the best thing to do may be to come to an agreement with him under which you will allow him to reduce what he owes you by the outstanding amount of the debt; that will constitute payment in full of the debt; and he will pay you the remaining balance.