A custom Operating System is a version of the same operating system with minor or major modifications. If you replace the ROM of your smartphone with a custom ROM, then you will be replacing the regular OS with a custom OS.