Repository: kmyk/online-judge-tools (python)

Issue:
download: no samples found in AtCoder tenka1_2014_qualA_e
`oj d` fails for <https://atcoder.jp/contests/tenka1-2014-quala/tasks/tenka1_2014_qualA_b> and <https://atcoder.jp/contests/tenka1-2014-quala/tasks/tenka1_2014_qualA_e> without making corrupted files.

嘘を出力するわけではないのであまり深刻ではないが、AtCoder 側の JavaScript はきちんと認識しているのでたぶん直せるし直しておくべき

``` console
$ oj d https://atcoder.jp/contests/tenka1-2014-quala/tasks/tenka1_2014_qualA_e       [x] problem recognized: AtCoderProblem.from_url('https://atcoder.jp/contests/tenka1-2014-quala/tasks/tenka1_2014_qualA_e'): https://atcoder.jp/contests/tenka1-2014-quala/tasks/tenka1_2014_qualA_e
[x] load cookie from: /home/ubuntu/.local/share/online-judge-tools/cookie.jar
[x] GET: https://atcoder.jp/contests/tenka1-2014-quala/tasks/tenka1_2014_qualA_e
[x] 200 OK
[x] save cookie to: /home/ubuntu/.local/share/online-judge-tools/cookie.jar
[x] append history to: /home/ubuntu/.cache/online-judge-tools/download-history.jsonl
```

Pull request description:
rewrite _find_sample_tags of AtCoder
close #618 

次をします:

- `if` `elif` `elif` の構造から単一の `for の構造へ変更する
- #618 のためのテストを足す
- (ついでに) コメントを足しておく

#618 の原因は `h3+pre.prettyprint` であるのに `h3+section pre` のための分岐に食われてしまって `h3` のために違う場所を探していたためだったようです。
同様の理由で `h3+pre.literal-block` なども (もしあれば) 検出に失敗します。

---

実装ですが、大きく構造を変えるのが楽なのでそうしました。
ついでに  `tag_plus` がよくない (無闇に `yield` してるのとか外側の変数 `pre` を読んでるのとか) も修正されました。バグ修正のついでの refactoring はよくないのですが、書き直しが多いので巻き込みました。

ところで、単なる `h3+section>pre` を足すと `<h3>Sample Output 1</h3><section><pre>13</pre>構成できる文字列は<pre><code>abc</code></pre>や……` で誤検出になります。これは意図的に回避しています。

(修正量が大きくて嫌な感じがしますが、元がまずかったのでしかたないやつだと思っています)

Patch:
diff --git a/onlinejudge/service/atcoder.py b/onlinejudge/service/atcoder.py
index 5dc8899..ce53444 100644
--- a/onlinejudge/service/atcoder.py
+++ b/onlinejudge/service/atcoder.py
@@ -635,35 +635,45 @@ class AtCoderProblemDetailedData(AtCoderProblemData):
 
     @classmethod
     def _find_sample_tags(cls, soup: bs4.BeautifulSoup) -> Iterator[Tuple[bs4.Tag, bs4.Tag]]:
-        def tag_plus(tag, expected_prv, expected_strings):
-            prv = tag.find_previous_sibling()
-            if prv and prv.name == expected_prv and prv.string and any(s in prv.string for s in expected_strings):
-                yield (pre, prv)
-
         expected_strings = ('入力例', '出力例', 'Sample Input', 'Sample Output')
 
-        if soup.find('pre', 'literal-block'):
-            # the first format: p+pre
-            # this format uses 'literal-block' in its page
-            # example: https://atcoder.jp/contests/utpc2011/tasks/utpc2011_1
-            for pre in soup.find_all('pre', 'literal-block'):
-                log.debug('pre tag: %s', str(pre))
-                yield from tag_plus(tag=pre, expected_prv='p', expected_strings=expected_strings)
+        def get_header(pre, tag, expected_tag_name):
+            if tag and tag.name == expected_tag_name and tag.string and any(s in tag.string for s in expected_strings):
+                return tag
+            return None
 
-        elif soup.find('pre', 'prettyprint linenums'):
-            # the second format: h3+section pre
-            # this format uses 'prettyprint linenums' in its page
-            # example: https://atcoder.jp/contests/abc003/tasks/abc003_4
-            for pre in soup.find_all('pre', 'prettyprint linenums'):
-                log.debug('pre tag: %s', str(pre))
-                yield from tag_plus(tag=pre.parent, expected_prv='h3', expected_strings=expected_strings)
+        for pre in soup.find(id='task-statement').find_all('pre'):
+            log.debug('pre tag: %s', str(pre))
 
-        elif soup.find('pre'):
-            # the third format: h3+pre
+            # the standard format: #task-statement h3+pre
+            # used by AtCoder's JavaScript, sometimes used with .prettyprint
             # example: https://atcoder.jp/contests/abc114/tasks/abc114_d
-            for pre in soup.find_all('pre'):
-                log.debug('pre tag: %s', str(pre))
-                yield from tag_plus(tag=pre, expected_prv='h3', expected_strings=expected_strings)
+            # NOTE: The AtCoder's JavaScript (at https://atcoder.jp/public/js/contest.js?v=201911110917 version) supports:
+            #     -   "#task-statement h3+pre" format for Copy buttons of <h3> and <pre> tags
+            #     -   "pre.prettyprint" format for Copy buttons of <pre> tags
+            h3 = get_header(pre, tag=pre.find_previous_sibling(), expected_tag_name='h3')
+            if h3:
+                yield (pre, h3)
+                continue
+
+            # a old format: #task-statement h3+section>pre.prettyprint
+            # partially supported by AtCoder's JavaScript
+            # NOTE: The relaxed format "#task-statement h3+section>pre" may cause false-positive.
+            # example: https://atcoder.jp/contests/abc003/tasks/abc003_4
+            if 'prettyprint' in pre.attrs.get('class', []) and pre.parent.name == 'section':
+                h3 = get_header(pre, tag=pre.parent.find_previous_sibling(), expected_tag_name='h3')
+                if h3:
+                    yield (pre, h3)
+                    continue
+
+            # a very old format: #task-statement p+pre.literal-block
+            # entirely unsupported by AtCoder's JavaScript
+            # example: https://atcoder.jp/contests/utpc2011/tasks/utpc2011_1
+            if 'literal-block' in pre.attrs.get('class', []):
+                p = get_header(pre, tag=pre.find_previous_sibling(), expected_tag_name='p')
+                if p:
+                    yield (pre, p)
+                    continue
 
     @classmethod
     def _parse_sample_cases(cls, soup: bs4.BeautifulSoup) -> List[onlinejudge.type.TestCase]:


Test patch:
diff --git a/tests/command_download_atcoder.py b/tests/command_download_atcoder.py
index 1907927..31f1de9 100644
--- a/tests/command_download_atcoder.py
+++ b/tests/command_download_atcoder.py
@@ -151,6 +151,23 @@ class DownloadAtCoderTest(unittest.TestCase):
             },
         ], type='json')
 
+    def test_call_download_atcoder_tenka1_2014_qualA_e(self):
+        # This problem uses an unusual HTML markup; see https://github.com/kmyk/online-judge-tools/issues/618
+        self.snippet_call_download('https://atcoder.jp/contests/tenka1-2014-quala/tasks/tenka1_2014_qualA_e', [
+            {
+                "input": "5 3\nAAB\nABB\nCDE\nFFH\nGHH\n2\n1 1\n2 3\n",
+                "output": "15\n7\n"
+            },
+            {
+                "input": "2 2\nAB\nBA\n2\n1 1\n2 1\n",
+                "output": "2\n2\n"
+            },
+            {
+                "input": "5 5\nAABAA\nACDEA\nAFGHA\nAIJKA\nAAAAA\n1\n3 1\n",
+                "output": "25\n"
+            },
+        ], type='json')
+
     def test_call_download_invalid_url(self):
         self.snippet_call_download_raises(requests.exceptions.HTTPError, 'http://abc001.contest.atcoder.jp/tasks/abc001_100')