Q:

Why isn't $D_\infty$ the set of symmetries of a circle? 

According to Wikipedia, "the infinite dihedral group Dih∞ is an infinite group with properties analogous to those of the finite dihedral groups."  However, it doesn't appear that this has anything to do with the symmetries of the circle, which surprised me, because that seems like the most natural generalization of the finite-order dihedral groups, which are sets of symmetries of regular polygons. 
Put another way, since regular polygons "approach" becoming a circle as the number of vertices, $n\to\infty$, it seems like $D_\infty$ should be the symmetries of the circle. 
So what kinds of symmetries does $D_\infty$ represent, and why was this group chosen for extension of the finite dihedral groups over the symmetries of the circle? ( Or are they related in some way I'm just not grasping?) 

A:

What you think of when taking $n$ to $\infty$ is this:

That is, you've got polygons inscribed in a common circle.  Note however that even in that model, the limit to infinity would not get the full circle, because only rational multiples of $2\pi$ will appear. 
But the limit that leads to $D_\infty$ is more like this:

And if you continue that to infinity, it's easy to see that the the sides will approach a straight line with equidistant points, and a rotation will end up as a translation on that line. 

A:

Although polygons "approach" the circle, the circle has quite a few more points than just the union of the vertices of all the $n$-gons inscribing it, and consequently it "should" have more symmetries.  The symmetry group (rotations and reflections) of the circle is known as $\text{O}(2, \mathbb{R})$ or sometimes just $\text{O}(2)$, which can be seen as the set of all orthogonal ($A^t = A^{-1}$) 2-by-2 matrices with real entries, under multiplication.  
Instead, you could think of $D_\infty$ as being the "limit" of the presentations for the finite dihedral groups, $$D_n = \langle r, s \mid r^n = s^2 = 1,\, srs = r^{-1} \rangle.$$
As $n$ goes to infinity, the statement that $r^n = 1$ simply drops out, because what "$r^n = 1$" is really saying is "$\text{ord}(r) = n$", and we write $\text{ord}(r) = \infty$ to mean $r$ has infinite order.  So
$$D_\infty = \langle r, s  \mid s^2 = 1,\, srs = r^{-1}\rangle$$
which some authors take as the definition of $D_\infty$.
Others define the infinite dihedral group as the isometries of $\mathbb{Z}$ as a subset of $\mathbb{R}$, in which case $x \mapsto x + 1$ can be thought of as $r$ (a "rotation") and $x \mapsto -x$ can be thought of as $s$ (a "reflection").  

A:

Finite dihedral groups, in general, are defined as particular quotients of the free group on two letters, with the associated normal subgroup being generated by the relations $r^n = f^2 = (rf)^2 = 1$.  The natural extension of this idea to the infinite case is to let $\langle r \rangle \cong \mathbb{Z}$ since the rotation subgroup is always cyclic, and the size of the rotation subgroup is the only thing differentiating all the various finite dihedral groups, structurally speaking. 
One problem with thinking about $D_{\infty}$ as the symmetry group of a circle is that the cardinalities simply don't match up.   As a quotient of a free group on two letters, $D_{\infty}$ is countably infinite.   On the other hand, the rotation subgroup of the group of symmetries of a circle is uncountably infinite as we can rotate $S^1$ by any angle $\theta \in [0, 2\pi)$.
Note that we can realize $D_{\infty}$ as the symmetry group of the set of integer points on a real number line: the rotation can be thought of as the translation $x \mapsto x \!  + \!  1$ and the reflection as reflection across the origin.   And this kind of makes sense given that the internal angle of a regular $n$-gon measures $\displaystyle 180- \frac{360}{n}$ degrees, and this approaches $180^{\circ}$ as $n \to \infty$.

