Repository: readthedocs/readthedocs.org

Issue:
Sitemap hreflang syntax invalid for regional language variants
The `hreflang` for regional variations in `sitemap.xml` follows the format of the URL language slug [generated by Sphinx](http://www.sphinx-doc.org/en/master/usage/configuration.html#confval-language), e.g. `zh_CN` for Chinese (China). This is invalid syntax for `hreflang` in a sitemap ([reference](https://en.wikipedia.org/wiki/Hreflang#Common_Mistakes)), where a hyphen must be used instead, e.g. `zh-CN`. Alternatively, it is also valid to define the script of the language instead, e.g. `zh-Hans` for Simplified Chinese. Some sort of library to perform this conversion, a lookup table or a user-facing option could help fix this.

Example from: https://docs.dash.org/sitemap.xml
```
<xhtml:link rel="alternate" hreflang="zh_CN" href="http://docs.dash.org/zh_CN/latest/"/>
```



Discussion:
The fact that we rely on sphinx for this make it hard to follow the standard way, there are other parts of the code that we should take care of too.
Hey @humitos can you tell me how should this be approached? And what needs to be changed to fix this?
@saadmk11 the `hreflang` label from the sitemap uses some incorrect codes. Those incorrect ones (for example, `zh_CN`) should be replaced for the correct ones (`zh-CN`). The linked document from the description has more detailed information.

Resolving patch:
diff --git a/readthedocs/core/views/serve.py b/readthedocs/core/views/serve.py
--- a/readthedocs/core/views/serve.py
+++ b/readthedocs/core/views/serve.py
@@ -356,6 +356,17 @@ def priorities_generator():
         priorities = [1, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2]
         yield from itertools.chain(priorities, itertools.repeat(0.1))
 
+    def hreflang_formatter(lang):
+        """
+        sitemap hreflang should follow correct format.
+
+        Use hyphen instead of underscore in language and country value.
+        ref: https://en.wikipedia.org/wiki/Hreflang#Common_Mistakes
+        """
+        if '_' in lang:
+            return lang.replace("_", "-")
+        return lang
+
     def changefreqs_generator():
         """
         Generator returning ``changefreq`` needed by sitemap.xml.
@@ -409,7 +420,7 @@ def changefreqs_generator():
                         private=False,
                     )
                     element['languages'].append({
-                        'hreflang': translation.language,
+                        'hreflang': hreflang_formatter(translation.language),
                         'href': href,
                     })
 


Test patch:
diff --git a/readthedocs/rtd_tests/tests/test_doc_serving.py b/readthedocs/rtd_tests/tests/test_doc_serving.py
--- a/readthedocs/rtd_tests/tests/test_doc_serving.py
+++ b/readthedocs/rtd_tests/tests/test_doc_serving.py
@@ -250,6 +250,13 @@ def test_sitemap_xml(self):
             main_language_project=self.public,
             language='translation-es'
         )
+        # sitemap hreflang should follow correct format.
+        # ref: https://en.wikipedia.org/wiki/Hreflang#Common_Mistakes
+        hreflang_test_translation_project = fixture.get(
+            Project,
+            main_language_project=self.public,
+            language='zh_CN'
+        )
         response = self.client.get(
             reverse('sitemap_xml'),
             HTTP_HOST='public.readthedocs.io',
@@ -287,3 +294,6 @@ def test_sitemap_xml(self):
                 private=False,
             ),
         )
+        # hreflang should use hyphen instead of underscore
+        # in language and country value. (zh_CN should be zh-CN)
+        self.assertContains(response, 'zh-CN')