Here's a brief description of how each functions. Casting directors screen actors, contact agents, put out casting calls, and arrange auditions. Some casting directors, but not all, belong to the Casting Society of America. There are several hundred casting directors working in the Los Angeles area.
Agents represent actors in various fields.
Within an agency, agents may specialize. Some agencies only represent actors in one or two fields. Agencies can represent from several to over a 1000 actors, depending on size of agency. All reputable agencies are franchise with the Screen Actors Guild, the actors union, and are regulated by the state. Many agents require actors to sign contracts, usually for a period of one year.