If you use someone else’s disabled badge and that badge holder is not in the car with you, either as a passenger or driver, then you have committed blue badge misuse, and that is a criminal offence under the current legislation.
Under Section 117 of the Road Traffic Regulation Act 1984, it is a criminal offence to misuse the Blue Badge and if you are convicted then you may get a criminal record and a possible fine of £1000. The Blue Badge may also be withdrawn.