質問: k*[(V_2-V_3) \times (V_1-V_3)]$. However, it doesn't really help. The second case is even more difficult to analyze. Second attempt consisted in solving this system by multiplication (scalar product or vector work) equations by appropriate vectors. However, I didn't really succeed.
回答: The algebra is not especially nice, but it is just algebra. This is rigid body rotation, taking point 3 as the origin of coordinates, so effectively $$\mathbf{r}_1=\mathbf{R}_1-\mathbf{R}_3, \qquad \mathbf{r}_2=\mathbf{R}_2-\mathbf{R}_3. $$ We start as you suggested, and abbreviate $$ \mathbf{v}_1=\mathbf{V}_1-\mathbf{V}_3, \qquad \mathbf{v}_2=\mathbf{V}_2-\mathbf{V}_3, $$ so that $$ \mathbf{v}_1 = \boldsymbol{\omega}\times\mathbf{r}_1, \qquad \mathbf{v}_2 = \boldsymbol{\omega}\times\mathbf{r}_2. $$ Now since the three points are not collinear, we can let $$ \boldsymbol{\omega} = a\,\mathbf{r}_1 + b\,\mathbf{r}_2 + c\, \mathbf{r}_1\times\mathbf{r}_2 $$ but we must remember that $\mathbf{r}_1$ and $\mathbf{r}_2$ will not in general be orthogonal. We can obtain $c$ directly, from either of the two equivalent equations \begin{align*} \mathbf{r}_2\cdot\mathbf{v}_1 &= \mathbf{r}_2\cdot\boldsymbol{\omega}\times\mathbf{r}_1 = \boldsymbol{\omega}\cdot\mathbf{r}_1\times\mathbf{r}_2 = c |\mathbf{r}_1\times\mathbf{r}_2|^2 \\ \mathbf{r}_1\cdot\mathbf{v}_2 &= \mathbf{r}_1\cdot\boldsymbol{\omega}\times\mathbf{r}_2 = -\boldsymbol{\omega}\cdot\mathbf{r}_1\times\mathbf{r}_2 = -c |\mathbf{r}_1\times\mathbf{r}_2|^2 \\ \Rightarrow\quad c&= \frac{\mathbf{r}_2\cdot\mathbf{v}_1}{|\mathbf{r}_1\times\mathbf{r}_2|^2} = -\frac{\mathbf{r}_1\cdot\mathbf{v}_2}{|\mathbf{r}_1\times\mathbf{r}_2|^2} \end{align*} where we took advantage of the properties of the scalar triple product. The other coefficients come from scalar products with $\mathbf{r}_1\times\mathbf{r}_2$. We use the general identity $$ (\mathbf{A}\times\mathbf{B})\cdot(\mathbf{C}\times\mathbf{D}) = (\mathbf{A}\cdot\mathbf{C})\,(\mathbf{B}\cdot\mathbf{D}) - (\mathbf{B}\cdot\mathbf{C})\,(\mathbf{A}\cdot\mathbf{D}) $$ and a special case of this, which we use, is $|\mathbf{r}_1\times\mathbf{r}_2|^2=|\mathbf{r}_1|^2|\mathbf{r}_2|^2-(\mathbf{r}_1\cdot\mathbf{r}_2)^2$. \begin{align*} \mathbf{r}_1\times\mathbf{r}_2 \cdot \mathbf{v}_2 &= (\mathbf{r}_1\times\mathbf{r}_2 ) \cdot (\boldsymbol{\omega}\times\mathbf{r}_2) \\ &= \left( a|\mathbf{r}_1|^2 + b(\mathbf{r}_1\cdot\mathbf{r}_2) \right)\, |\mathbf{r}_2|^2- \left( a(\mathbf{r}_1\cdot\mathbf{r}_2) + b|\mathbf{r}_2|^2 \right)\, (\mathbf{r}_1\cdot\mathbf{r}_2) \\ &= a |\mathbf{r}_1\times\mathbf{r}_2|^2 \\ \Rightarrow\quad a&=\frac{\mathbf{r}_1\times\mathbf{r}_2 \cdot \mathbf{v}_2 }{|\mathbf{r}_1\times\mathbf{r}_2|^2} \\ \mathbf{r}_1\times\mathbf{r}_2 \cdot \mathbf{v}_1 &= (\mathbf{r}_1\times\mathbf{r}_2 ) \cdot (\boldsymbol{\omega}\times\mathbf{r}_1) \\ &= \left( a|\mathbf{r}_1|^2 + b(\mathbf{r}_1\cdot\mathbf{r}_2) \right)\,(\mathbf{r}_1\cdot\mathbf{r}_2) - \left( a(\mathbf{r}_1\cdot\mathbf{r}_2) + b|\mathbf{r}_2|^2 \right) \, |\mathbf{r}_1|^2 \\ &= -b |\mathbf{r}_1\times\mathbf{r}_2|^2 \\ \Rightarrow\quad b &=-\frac{\mathbf{r}_1\times\mathbf{r}_2 \cdot \mathbf{v}_1}{|\mathbf{r}_1\times\mathbf{r}_2|^2} \end{align*} I hope I haven't made any slips, you should definitely check!