Create broken-linkcheck.yml
State: closed

Created a Github Action to run the Sphinx linkcheck monthly. 

- [x] closes #xxxx (Replace xxxx with the GitHub issue number) #45409
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
Could you make this workflow run in this PR to check the output?
> Could you make this workflow run in this PR to check the output?

Could you tell me how to do that ? Do I need to change the yaml code ?
Yes see https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#on or other workflows in this repo
I changed it to on:  `workflow_dispatch`. But still I don't see any `run workflow` button . [manually-running-a-workflow](https://docs.github.com/en/actions/using-workflows/manually-running-a-workflow)
Is it the proper way to run the workflow immediately ? 
@pmeier could you see this please ?
Linkcheck failed . It says 

```

File "/home/runner/work/pandas/pandas/doc/make.py", line 23, in <module>
    import docutils
ModuleNotFoundError: No module named 'docutils'

```
Sorry, you also need to put 

https://github.com/pandas-dev/pandas/blob/89bd5695df3ccf5b9186b4645118656656831593/.github/workflows/docbuild-and-upload.yml#L33-L37

inside the workflow to automatically activate the conda env.
@pmeier ,  there are many broken links. How to fix them ? 
The linkcheck is showing proper output , then why the ci is showing it as failure ?
@mroeschke , could you look into the pr if anything else to add?
> You could also add ignores for all broken links for now and chip away at the ignore list later on.

But we have to create a new issue (and delete the old one) reporting all the errors and label and Docs and Good First Issue (no. 3)
 https://github.com/pandas-dev/pandas/issues/45409#issue-1105272826
@pmeier , could you tell me how to write the script for no 3 problem (new issue (and delete the old one) Docs and Good First Issue etc etc) ?
 And what about this https://github.com/pandas-dev/pandas/pull/55246#discussion_r1339864162 ? The anchor issues ?

> The ignore list can likely be build programmatically from the output of the CI job.

Do I need to put that script into [conf.py](https://github.com/pandas-dev/pandas/blob/main/doc/source/conf.py) too? where shoould I put that into the conf.py file ?
@pmeier 
@pmeier , I have written the ignored list script in the drop-down. Should I create another PR for it or should I attach it in here?

<Details>

```
import re
import requests

# Fetch the content of the ci file
pipeline_url = ("https://pipelinesghubeus22.actions.githubusercontent.com/"
                "xZyE9jtmkxWlfCAbyu1SHPJOlsa2huNFYcxohSTomy6EbdNZT9/"
                "_apis/pipelines/1/runs/531865/signedlogcontent/2?"
                "urlExpires=2023-10-08T10%3A02%3A18.0563894Z&"
                "urlSigningMethod=HMACV1&"
                "urlSignature=9icPkVbCE2Ya0M5%2FY03N8fkFuDfYBn%2F"
                "DyJ93o2R4%2BWk%3D")

response = requests.get(pipeline_url)
ignore_patterns = response.text.splitlines()

# List to store broken links
broken_links = []

# Ignore links based on patterns from ci file
for pattern in ignore_patterns:
    # Compile the regular expression pattern
    regex_pattern = re.compile(pattern)

    # Check if any broken link matches the pattern
    ignored_links = [link for link in broken_links if regex_pattern.search(link)]


    print(f"Ignored links matching pattern {pattern}:")
    print(ignored_links)
```

</Details>
@ggold7046 As is the script is not executable, since the link is expired. In any case, this script is just for you to automatically build the ignore list that you'll need to put into `conf.py`. Looking at the CI output, we only have roughly 30 broken links. With that number it is still feasible to do it manually.
I can not see the linkcheck.txt file in the ./doc directory.
> Looking at the CI output, we only have roughly 30 broken links. With that number it is still feasible to do it manually.

I can only see a single 404 from the linkcheck output. 


> I can not see the linkcheck.txt file in the ./doc directory.

This file only exists in the CI runner and won't be uploaded or the like.

> I can only see a single 404 from the linkcheck output.

You can see all failures marked as "broken" here: https://github.com/pandas-dev/pandas/actions/runs/6420191351/job/17431740319?pr=55246#step:6:9
@pmeier , I have opened a new issue for this broken links. Let's take the first issue.

`(whatsnew/v0.13.1: line   22) broken    http://scatterci.github.io/pydata/pandas - 404 Client Error: Not Found for url: http://scatterci.github.io/pydata/pandas`

Should I ask someone to provide me with the correct link and put that correct link into the doc ? How would I find out which page of the doc contains this erroneous link ? 
> I have opened a new issue for this broken links

#55455 for xref.

> How would I find out which page of the doc contains this erroneous link ?

The first part of the error tells you the location

`(whatsnew/v0.13.1: line   22)`

The path is relative to [`doc/source`](https://github.com/pandas-dev/pandas/tree/main/doc/source).

https://github.com/pandas-dev/pandas/blob/2a65fdd227734cbda5d8e33171e857ab4aaeb9d5/doc/source/whatsnew/v0.13.1.rst?plain=1#L22

> Should I ask someone to provide me with the correct link and put that correct link into the doc ?

Well, it is up to you (and other contributors) that want to take this up to find out what is the new correct link. Maybe the resource we originally were linking to just moved. Maybe it was removed.

Let's take the example above. The broken link comes from a release note from 2014 and links to a CI provider. Googling "scatterci" yields no useful results. In addition, `pandas` no longer uses scatterci as CI provider. Searching through the pandas documentation also yields [only the result that we have above already](https://pandas.pydata.org/docs/search.html?q=scatterci).

Meaning, it is probably safe to just remove the link all together. 

I'll just say it again. I would just add all broken links to an ignore list as part of this PR and with follow-up PRs start fixing links and thus removing things from the ignore list. Otherwise, this PR will open for a long time.
https://github.com/pandas-dev/pandas/actions/runs/6500122547/job/17654854642?pr=55246#step:5:149

>  ```
>  Exception occurred:
>   File "/home/runner/micromamba/envs/test/lib/python3.10/sre_parse.py", line 843, in _parse
>     raise source.error("missing ), unterminated subpattern",
> ```

And this is why I [suggested](https://github.com/pandas-dev/pandas/pull/55246#discussion_r1354307474) wrapping everything in `re.escape`. Basically what happened is that something in the ignore list included special characters that were picked up by the regex engine. Since we don't want to do regex matching at this point, we should just match the exact version of the link, hence escaping it.
@pmeier 
> NameError: name 're' is not defined

@pmeier , I have imported the `re` module.

```
name: Linkcheck
on:
    workflow_dispatch: 
    # Enable manual triggering
    schedule:
    # Run monthly on the 1st day of the month
        - cron: '0 0 1 * *'
```
The cron tab is showing this error `Runs at 00:00, on day 1 of the month. Actions schedules run at most every 5 minutes using UTC time.` That is why the yaml file ci is also failing.




> The cron tab is showing this error `Runs at 00:00, on day 1 of the month. Actions schedules run at most every 5 minutes using UTC time.` That is why the yaml file ci is also failing.

And how to rectify this ?
> And how to rectify this ?

Could you post a (non-expiring) link to where you got this from? I only see a trailing whitespace issue on the CI builds.
> Could you post a (non-expiring) link to where you got this from?

![Screenshot_2023-19](https://github.com/pandas-dev/pandas/assets/98327736/45b61286-156b-4dae-a1db-dc3bd574b0cb)

There is a blue underline in cron. When hovering it shows the error and showing this [Link](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onschedule)

![cron](https://github.com/pandas-dev/pandas/assets/98327736/b3e6420c-d4bb-425b-b03d-82a3ed839fdf)



@pmeier , are you available on pandas-dev slack ?
@pmeier , is this PR complete apart from the two ci fails ? 

I corrected a line which exceeded more than 88 characters in the conf.py file, but still the pre-coomit.ci is failing.  I couldn't understand any more errors. Neither I understood the circle ci fail.
pre-commit.ci autofix
> is this PR complete apart from the two ci fails ?

From my side, yes. But now we need a maintainer to review and approve.

> I corrected a line which exceeded more than 88 characters in the conf.py file, but still the pre-coomit.ci is failing.

You need to comment "pre-commit.ci autofix" for the fix to be applied. I tried in https://github.com/pandas-dev/pandas/pull/55246#issuecomment-1768298310, but I don't have permissions. It is basically complaining that the imports in `conf.py` are not ordered correctly.

> Neither I understood the circle ci fail.

This is most likely unrelated to this PR. I'd wager a guess that if you update your branch from the `main` branch, the issue will go away.
pre-commit.ci autofix