No, the Panel Physician does not decide on your admissibility or inadmissibility. The Panel Physician only gathers medical information and adds it to your medical records during the physical examination. After completing the blood work and the X-rays, the panel physician will submit your case to your immigration application.
The immigration officers will then decide if you are medically admissible or inadmissible. They can return your medical case and ask for further tests.
Check our guide on the role of a panel physician for better understanding. Also, our resources on what happens if you fail your IME can help you understand IME further.