Yes. One of the main disadvantages of using a kernel binary is that you do not have any control with its built-in features. This means that, unlike a source-based kernel, a kernel binary will not contain any hardware-specific optimizations and support.
This means that if you intend on using exotic pieces of hardware with Ubuntu. You will need compile your kernel from scratch.