Subfield: Geometry


Question: Let $n \geqslant 3$ be an integer. Two regular $n$-gons $\mathcal{A}$ and $\mathcal{B}$ are given in the plane. Prove that the vertices of $\mathcal{A}$ that lie inside $\mathcal{B}$ or on its boundary are consecutive.

(That is, prove that there exists a line separating those vertices of $\mathcal{A}$ that lie inside $\mathcal{B}$ or on its boundary from the other vertices of $\mathcal{A}$.)

Solution:
By a polygon we always mean its interior together with its boundary.

We start with finding a regular $n$-gon $\mathcal{C}$ which $(i)$ is inscribed into $\mathcal{B}$ (that is, all vertices of $\mathcal{C}$ lie on the perimeter of $\mathcal{B}$ ); and $($ ii) is either a translation of $\mathcal{A}$, or a homothetic image of $\mathcal{A}$ with a positive factor.

Such a polygon may be constructed as follows. Let $O_{A}$ and $O_{B}$ be the centers of $\mathcal{A}$ and $\mathcal{B}$, respectively, and let $A$ be an arbitrary vertex of $\mathcal{A}$. Let $\overrightarrow{O_{B} C}$ be the vector co-directional to $\overrightarrow{O_{A} A}$, with $C$ lying on the perimeter of $\mathcal{B}$. The rotations of $C$ around $O_{B}$ by multiples of $2 \pi / n$ form the required polygon. Indeed, it is regular, inscribed into $\mathcal{B}$ (due to the rotational symmetry of $\mathcal{B}$ ), and finally the translation/homothety mapping $\overrightarrow{O_{A} A}$ to $\overrightarrow{O_{B} C}$ maps $\mathcal{A}$ to $\mathcal{C}$.

Now we separate two cases.

<img_3389>

Construction of $\mathcal{C}$

<img_3994>

Case 1: Translation

Case 1: $\mathcal{C}$ is a translation of $\mathcal{A}$ by a vector $\vec{v}$.

Denote by $t$ the translation transform by vector $\vec{v}$. We need to prove that the vertices of $\mathcal{C}$ which stay in $\mathcal{B}$ under $t$ are consecutive. To visualize the argument, we refer the plane to Cartesian coordinates so that the $x$-axis is co-directional with $\vec{v}$. This way, the notions of right/left and top/bottom are also introduced, according to the $x$ - and $y$-coordinates, respectively.

Let $B_{\mathrm{T}}$ and $B_{\mathrm{B}}$ be the top and the bottom vertices of $\mathcal{B}$ (if several vertices are extremal, we take the rightmost of them). They split the perimeter of $\mathcal{B}$ into the right part $\mathcal{B}_{\mathrm{R}}$ and the left part $\mathcal{B}_{\mathrm{L}}$ (the vertices $B_{\mathrm{T}}$ and $B_{\mathrm{B}}$ are assumed to lie in both parts); each part forms a connected subset of the perimeter of $\mathcal{B}$. So the vertices of $\mathcal{C}$ are also split into two parts $\mathcal{C}_{\mathrm{L}} \subset \mathcal{B}_{\mathrm{L}}$ and $\mathcal{C}_{\mathrm{R}} \subset \mathcal{B}_{\mathrm{R}}$, each of which consists of consecutive vertices.

Now, all the points in $\mathcal{B}_{\mathrm{R}}$ (and hence in $\mathcal{C}_{\mathrm{R}}$ ) move out from $\mathcal{B}$ under $t$, since they are the rightmost points of $\mathcal{B}$ on the corresponding horizontal lines. It remains to prove that the vertices of $\mathcal{C}_{\mathrm{L}}$ which stay in $\mathcal{B}$ under $t$ are consecutive.

For this purpose, let $C_{1}, C_{2}$, and $C_{3}$ be three vertices in $\mathcal{C}_{\mathrm{L}}$ such that $C_{2}$ is between $C_{1}$ and $C_{3}$, and $t\left(C_{1}\right)$ and $t\left(C_{3}\right)$ lie in $\mathcal{B}$; we need to prove that $t\left(C_{2}\right) \in \mathcal{B}$ as well. Let $A_{i}=t\left(C_{i}\right)$. The line through $C_{2}$ parallel to $\vec{v}$ crosses the segment $C_{1} C_{3}$ to the right of $C_{2}$; this means that this line crosses $A_{1} A_{3}$ to the right of $A_{2}$, so $A_{2}$ lies inside the triangle $A_{1} C_{2} A_{3}$ which is contained in $\mathcal{B}$. This yields the desired result.

Case 2: $\mathcal{C}$ is a homothetic image of $\mathcal{A}$ centered at $X$ with factor $k>0$.



Denote by $h$ the homothety mapping $\mathcal{C}$ to $\mathcal{A}$. We need now to prove that the vertices of $\mathcal{C}$ which stay in $\mathcal{B}$ after applying $h$ are consecutive. If $X \in \mathcal{B}$, the claim is easy. Indeed, if $k<1$, then the vertices of $\mathcal{A}$ lie on the segments of the form $X C$ ( $C$ being a vertex of $\mathcal{C})$ which lie in $\mathcal{B}$. If $k>1$, then the vertices of $\mathcal{A}$ lie on the extensions of such segments $X C$ beyond $C$, and almost all these extensions lie outside $\mathcal{B}$. The exceptions may occur only in case when $X$ lies on the boundary of $\mathcal{B}$, and they may cause one or two vertices of $\mathcal{A}$ stay on the boundary of $\mathcal{B}$. But even in this case those vertices are still consecutive.

So, from now on we assume that $X \notin \mathcal{B}$.

Now, there are two vertices $B_{\mathrm{T}}$ and $\mathcal{B}_{\mathrm{B}}$ of $\mathcal{B}$ such that $\mathcal{B}$ is contained in the angle $\angle B_{\mathrm{T}} X B_{\mathrm{B}}$; if there are several options, say, for $B_{\mathrm{T}}$, then we choose the farthest one from $X$ if $k>1$, and the nearest one if $k<1$. For the visualization purposes, we refer the plane to Cartesian coordinates so that the $y$-axis is co-directional with $\overrightarrow{B_{\mathrm{B}} B_{\mathrm{T}}}$, and $X$ lies to the left of the line $B_{\mathrm{T}} B_{\mathrm{B}}$. Again, the perimeter of $\mathcal{B}$ is split by $B_{\mathrm{T}}$ and $B_{\mathrm{B}}$ into the right part $\mathcal{B}_{\mathrm{R}}$ and the left part $\mathcal{B}_{\mathrm{L}}$, and the set of vertices of $\mathcal{C}$ is split into two subsets $\mathcal{C}_{\mathrm{R}} \subset \mathcal{B}_{\mathrm{R}}$ and $\mathcal{C}_{\mathrm{L}} \subset \mathcal{B}_{\mathrm{L}}$.

<img_3597>

Case $2, X$ inside $\mathcal{B}$

<img_3767>

Subcase 2.1: $k>1$

Subcase 2.1: $k>1$.

In this subcase, all points from $\mathcal{B}_{\mathrm{R}}$ (and hence from $\mathcal{C}_{\mathrm{R}}$ ) move out from $\mathcal{B}$ under $h$, because they are the farthest points of $\mathcal{B}$ on the corresponding rays emanated from $X$. It remains to prove that the vertices of $\mathcal{C}_{\mathrm{L}}$ which stay in $\mathcal{B}$ under $h$ are consecutive.

Again, let $C_{1}, C_{2}, C_{3}$ be three vertices in $\mathcal{C}_{\mathrm{L}}$ such that $C_{2}$ is between $C_{1}$ and $C_{3}$, and $h\left(C_{1}\right)$ and $h\left(C_{3}\right)$ lie in $\mathcal{B}$. Let $A_{i}=h\left(C_{i}\right)$. Then the ray $X C_{2}$ crosses the segment $C_{1} C_{3}$ beyond $C_{2}$, so this ray crosses $A_{1} A_{3}$ beyond $A_{2}$; this implies that $A_{2}$ lies in the triangle $A_{1} C_{2} A_{3}$, which is contained in $\mathcal{B}$.

<img_3986>

Subcase $2.2: k<1$

Subcase 2.2: $k<1$.

This case is completely similar to the previous one. All points from $\mathcal{B}_{\mathrm{L}}$ (and hence from $\mathcal{C}_{\mathrm{L}}$ move out from $\mathcal{B}$ under $h$, because they are the nearest points of $\mathcal{B}$ on the corresponding



rays emanated from $X$. Assume that $C_{1}, C_{2}$, and $C_{3}$ are three vertices in $\mathcal{C}_{\mathrm{R}}$ such that $C_{2}$ lies between $C_{1}$ and $C_{3}$, and $h\left(C_{1}\right)$ and $h\left(C_{3}\right)$ lie in $\mathcal{B}$; let $A_{i}=h\left(C_{i}\right)$. Then $A_{2}$ lies on the segment $X C_{2}$, and the segments $X A_{2}$ and $A_{1} A_{3}$ cross each other. Thus $A_{2}$ lies in the triangle $A_{1} C_{2} A_{3}$, which is contained in $\mathcal{B}$.
By a polygon we always mean its interior together with its boundary.

Let $O_{A}$ and $O_{B}$ be the centers of $\mathcal{A}$ and $\mathcal{B}$, respectively. Denote $[n]=\{1,2, \ldots, n\}$.

We start with introducing appropriate enumerations and notations. Enumerate the sidelines of $\mathcal{B}$ clockwise as $\ell_{1}, \ell_{2}, \ldots, \ell_{n}$. Denote by $\mathcal{H}_{i}$ the half-plane of $\ell_{i}$ that contains $\mathcal{B}\left(\mathcal{H}_{i}\right.$ is assumed to contain $\ell_{i}$ ); by $B_{i}$ the midpoint of the side belonging to $\ell_{i}$; and finally denote $\overrightarrow{b_{i}}=\overrightarrow{B_{i} O_{B}}$. (As usual, the numbering is cyclic modulo $n$, so $\ell_{n+i}=\ell_{i}$ etc.)

Now, choose a vertex $A_{1}$ of $\mathcal{A}$ such that the vector $\overrightarrow{O_{A} A_{1}}$ points "mostly outside $\mathcal{H}_{1}$ "; strictly speaking, this means that the scalar product $\left\langle\overrightarrow{O_{A} A_{1}}, \overrightarrow{b_{1}}\right\rangle$ is minimal. Starting from $A_{1}$, enumerate the vertices of $\mathcal{A}$ clockwise as $A_{1}, A_{2}, \ldots, A_{n}$; by the rotational symmetry, the choice of $A_{1}$ yields that the vector $\overrightarrow{O_{A} A_{i}}$ points "mostly outside $\mathcal{H}_{i}$ ", i.e.,

$$
\left\langle\overrightarrow{O_{A} A_{i}}, \overrightarrow{b_{i}}\right\rangle=\min _{j \in[n]}\left\langle\overrightarrow{O_{A} A_{j}}, \overrightarrow{b_{i}}\right\rangle
\tag{1}
$$

<img_3824>

Enumerations and notations

We intend to reformulate the problem in more combinatorial terms, for which purpose we introduce the following notion. Say that a subset $I \subseteq[n]$ is connected if the elements of this set are consecutive in the cyclic order (in other words, if we join each $i$ with $i+1 \bmod n$ by an edge, this subset is connected in the usual graph sense). Clearly, the union of two connected subsets sharing at least one element is connected too. Next, for any half-plane $\mathcal{H}$ the indices of vertices of, say, $\mathcal{A}$ that lie in $\mathcal{H}$ form a connected set.

To access the problem, we denote

$$
M=\left\{j \in[n]: A_{j} \notin \mathcal{B}\right\}, \quad M_{i}=\left\{j \in[n]: A_{j} \notin \mathcal{H}_{i}\right\} \quad \text { for } i \in[n]
$$

We need to prove that $[n] \backslash M$ is connected, which is equivalent to $M$ being connected. On the other hand, since $\mathcal{B}=\bigcap_{i \in[n]} \mathcal{H}_{i}$, we have $M=\bigcup_{i \in[n]} M_{i}$, where the sets $M_{i}$ are easier to investigate. We will utilize the following properties of these sets; the first one holds by the definition of $M_{i}$, along with the above remark.



<img_3628>

The sets $M_{i}$

Property 1: Each set $M_{i}$ is connected.

Property 2: If $M_{i}$ is nonempty, then $i \in M_{i}$.

Proof. Indeed, we have

$$
j \in M_{i} \Longleftrightarrow A_{j} \notin \mathcal{H}_{i} \Longleftrightarrow\left\langle\overrightarrow{B_{i} A_{j}}, \overrightarrow{b_{i}}\right\rangle<0 \Longleftrightarrow\left\langle\overrightarrow{O_{A} A_{j}}, \overrightarrow{b_{i}}\right\rangle<\left\langle\overrightarrow{O_{A} B_{i}}, \overrightarrow{b_{i}}\right\rangle .
\tag{2}
$$

The right-hand part of the last inequality does not depend on $j$. Therefore, if some $j$ lies in $M_{i}$, then by (1) so does $i$.

In view of Property 2, it is useful to define the set

$$
M^{\prime}=\left\{i \in[n]: i \in M_{i}\right\}=\left\{i \in[n]: M_{i} \neq \varnothing\right\}
$$

Property 3: The set $M^{\prime}$ is connected.

Proof. To prove this property, we proceed on with the investigation started in (2) to write

$$
i \in M^{\prime} \Longleftrightarrow A_{i} \in M_{i} \Longleftrightarrow\left\langle\overrightarrow{B_{i} A_{i}}, \overrightarrow{b_{i}}\right\rangle<0 \Longleftrightarrow\left\langle\overrightarrow{O_{B} O_{A}}, \overrightarrow{b_{i}}\right\rangle<\left\langle\overrightarrow{O_{B} B_{i}}, \overrightarrow{b_{i}}\right\rangle+\left\langle\overrightarrow{A_{i} O_{A}}, \overrightarrow{b_{i}}\right\rangle
$$

The right-hand part of the obtained inequality does not depend on $i$, due to the rotational symmetry; denote its constant value by $\mu$. Thus, $i \in M^{\prime}$ if and only if $\left\langle\overrightarrow{O_{B} O_{A}}, \overrightarrow{b_{i}}\right\rangle<\mu$. This condition is in turn equivalent to the fact that $B_{i}$ lies in a certain (open) half-plane whose boundary line is orthogonal to $O_{B} O_{A}$; thus, it defines a connected set.

Now we can finish the solution. Since $M^{\prime} \subseteq M$, we have

$$
M=\bigcup_{i \in[n]} M_{i}=M^{\prime} \cup \bigcup_{i \in[n]} M_{i}
$$

so $M$ can be obtained from $M^{\prime}$ by adding all the sets $M_{i}$ one by one. All these sets are connected, and each nonempty $M_{i}$ contains an element of $M^{\prime}$ (namely, $i$ ). Thus their union is also connected.