You just plug the modem into the serial port. Such an operating system would then know that you have a modem on a certain port and would also know the id number. If it's a software modem, it could try to locate a driver for it. It could also tell application programs what port your modem is on. Some programs can probe for a modem on various ports.
Modems use straight through cable, with no pins crossed over.
Most computer stores should have this. Make sure you get the correct gender and number of pins. Hook up your modem to one of your serial ports. If you are willing to accept the default IRQ and IO address of the port you connect it to, then you are ready to start your communication program and configure the modem itself. An internal modem is installed in a PC by taking off the cover of the PC and inserting the modem card into a vacant slot on the motherboard.