System

system, n. the set of rules that a role-playing game uses.
----
Addendum: Systems are the skeleton of an RPG, upon which the setting and stories are laid down upon. Each system is generally unique to each game, unless they are using a generic system such as d20, or differing games from the same publisher such as GURPS or Hero System.
Addendum #2: Many systems revolve around a specific die for most of the rolls and tests made in the game, either using the other dice for secondary rolls (such as with the d20 system that used the twenty-sided die for most skill checks and the others generally for damage), or eliminates the other dice altogether (such as with the core system of the World of Darkness, which exclusively use ten-sided dice). Gamers often refer to these systems by their die type, i.e. a "d10 system", "d6 system", etc.
Addendum #3: Systems often have their defenders and detractors that range into Mac/PC arguments.