In mathematics, and in particular the study of dynamical systems, the idea of stable and unstable sets or stable and unstable manifolds give a formal mathematical definition to the general notions embodied in the idea of an attractor or repellor. In the case of hyperbolic dynamics, the corresponding notion is that of the hyperbolic set.

Physical example 
The gravitational tidal forces acting on the rings of Saturn provide an easy-to-visualize physical example. The tidal forces flatten the ring into the equatorial plane, even as they stretch it out in the radial direction. Imagining the rings to be sand or gravel particles ("dust") in orbit around Saturn, the tidal forces are such that any perturbations that push particles above or below the equatorial plane results in that particle feeling a restoring force, pushing it back into the plane. Particles effectively oscillate in a harmonic well, damped by collisions. The stable direction is perpendicular to the ring. The unstable direction is along any radius, where forces stretch and pull particles apart. Two particles that start very near each other in phase space will experience radial forces causing them to diverge, radially. These forces have a positive Lyapunov exponent; the trajectories lie on a hyperbolic manifold, and the movement of particles is essentially chaotic, wandering through the rings. The center manifold is tangential to the rings, with particles experiencing neither compression nor stretching. This allows second-order gravitational forces to dominate, and so particles can be entrained by moons or moonlets in the rings, phase locking to them. The gravitational forces of the moons effectively provide a regularly repeating small kick, each time around the orbit, akin to a kicked rotor, such as found in a phase-locked loop. 

The discrete-time motion of particles in the ring can be approximated by the Poincaré map. The map effectively provides the transfer matrix of the system. The eigenvector associated with the largest eigenvalue of the matrix is the Frobenius–Perron eigenvector, which is also the invariant measure, i.e the actual density of the particles in the ring. All other eigenvectors of the transfer matrix have smaller eigenvalues, and correspond to decaying modes.

Definition
The following provides a definition for the case of a system that is either an iterated function or has discrete-time dynamics. Similar notions apply for systems whose time evolution is given by a flow.

Let  be a topological space, and  a homeomorphism. If  is a fixed point for , the stable set of  is defined by

and the unstable set of  is defined by

Here,  denotes the inverse of the function , i.e.
, where  is the identity map on .

If  is a periodic point of least period , then it is a fixed point of , and the stable and unstable sets of  are defined by

and

Given a neighborhood  of , the local stable and unstable sets of  are defined by

and

If  is metrizable, we can define the stable and unstable sets for any point by

and

where  is a metric for . This definition clearly coincides with the previous one when  is a periodic point.

Suppose now that  is a compact smooth manifold, and  is a  diffeomorphism, . If  is a hyperbolic periodic point, the stable manifold theorem assures that for some neighborhood  of , the local stable and unstable sets are   embedded disks, whose tangent spaces at  are  and  (the stable and unstable spaces of ), respectively; moreover, they vary continuously (in a certain sense) in a neighborhood of  in the  topology of  (the space of all  diffeomorphisms from  to itself). Finally, the stable and unstable sets are  injectively immersed disks. This is why they are commonly called stable and unstable manifolds. This result is also valid for nonperiodic points, as long as they lie in some hyperbolic set (stable manifold theorem for hyperbolic sets).

Remark
If  is a (finite-dimensional) vector space and  an isomorphism, its stable and unstable sets are called stable space and unstable space, respectively.

See also
 Invariant manifold
 Center manifold
 Limit set
 Julia set
 Slow manifold
 Inertial manifold
 Normally hyperbolic invariant manifold
 Lagrangian coherent structure

References

Limit sets
Dynamical systems
Manifolds