The loss of viewers continued into season 12, which saw the show hitting a number of series low in the 18-49 demo