Repository: cli/cli
Pull request #10767: Fix multi pages search for gh search
Author: login: leudz
Labels: name: external
description: pull request originating outside of the CLI core team
Created: 2025-04-11T01:54:05Z | Merged: 2025-04-16T20:57:12Z

This fixes how `gh search` retrieves items when more than 100 items are requested and not a multiple of 100.

When using pagination, two variables are used: page and perPage. Pages can contain a maximum of 100 items.
The previous implementation tried to save the API some work by requesting less items for the last page.
But it didn't handle pagination correctly and ended up requesting some items a second time.

For example if 150 items are queried, the first page returns items 0->99.
Then it would try to query 50 items by asking for page 2 with 50 per page.
But this would return items 50->99 which isn't what we want.

This new version instead picks a fixed amount per page, in this case 100.
The first page returns items 0->99 and the second 100->199.
Then we discard items 150->199.

When less than 100 items are requested there is no change, we only request what we need.

----

In theory it's possible to both save the API some work and keep track of pagination but that would be more complex.

Fixes #9749

Linked issue #9749: `gh search` commands returns duplicate data when `--limit` greater than 100 but not a multiple of 100
```console
% gh --version   
gh version 2.58.0 (2024-10-01)
https://github.com/cli/cli/releases/tag/v2.58.0
% gh -R brave/brave-core search prs --limit 256 --merged --merged-at ">$(date -I -d '1 month ago')" --base=master --json number -q '.[].number'|wc -l
256
% gh -R brave/brave-core search prs --limit 256 --merged --merged-at ">$(date -I -d '1 month ago')" --base=master --json number -q '.[].number'|sort -u|wc -l
200
% gh -R brave/brave-core search prs --limit 275 --merged --merged-at ">$(date -I -d '1 month ago')" --base=master --json number -q '.[].number'|wc -l
275
% gh -R brave/brave-core search prs --limit 275 --merged --merged-at ">$(date -I -d '1 month ago')" --base=master --json number -q '.[].number'|sort -u|wc -l
225
% gh -R brave/brave-core search prs --limit 300 --merged --merged-at ">$(date -I -d '1 month ago')" --base=master --json number -q '.[].number'|wc -l        
241
% gh -R brave/brave-core search prs --limit 300 --merged --merged-at ">$(date -I -d '1 month ago')" --base=master --json number -q '.[].number'|sort -u|wc -l
241
%
```
Linked issue comments: user: login: andyfeller
created_at: 2024-10-16T14:23:34Z
body: Thanks for opening this up, @wknapik, that is some weird behavior! 🤔 I'd like to share my theory and propose a few alternatives for reproducing this.

Essentially, I'm not sure `wc -l` or `sort -u | wc -l` are appropriate ways to manipulate and understand this data.  It would help to preserve the output in files that we can compare and understand why they are differing.

### Let's try to reproduce this

Here are some alterations I like to propose understanding what's going on:

1. Let's pick a concrete date to look for merges

   _Let's base this on the date you opened this issue: `2024-09-14`_

1. Let's consider preserving the output from the command into a file so we can compare how they are different

1. Let's use `jq` to count the length of the arrays rather than `wc -l`

   _We have to use `jq` rather than `--jq` if we want to preserve the output, should only require: `... | jq '. | length'`

The result looks something like this:

```shell
# We should be able to see deviation in GitHub API responses after 10 attempts with the same arguments
for i in {1..10}; do \
    echo "Attempt $i";
    gh search prs --limit 256 --merged --merged-at ">2024-09-14" --base master  --repo brave/brave-core --json number > "cli-9749-$i.json";
done

# This is a more accurate way to look purely at the number of items
find . -name "cli-9749-*.json" -print -exec jq '. | length' {} \;

for i in {2..10}; do \
    diff <(jq 'sort' cli-9749-1.json) <(jq 'sort' "cli-9749-$i.json");
done
```

In theory, this will check the number of results and the actual contents.

### How did this work out?

**10 attempts all consistent both number of results and actual contents**

```shell
andyfeller@Andys-MBP:~ $ gh version
gh version 2.58.0 (2024-10-01)
https://github.com/cli/cli/releases/tag/v2.58.0


andyfeller@Andys-MBP:~ $ for i in {1..10}; do \
    echo "Attempt $i";
    gh search prs --limit 256 --merged --merged-at ">2024-09-14" --base master  --repo brave/brave-core --json number > "cli-9749-$i.json";
done
Attempt 1
Attempt 2
Attempt 3
Attempt 4
Attempt 5
Attempt 6
Attempt 7
Attempt 8
Attempt 9
Attempt 10


andyfeller@Andys-MBP:~$ find . -name "cli-9749-*.json" -print -exec jq '. | length' {} \;

./cli-9749-2.json
256
./cli-9749-3.json
256
./cli-9749-8.json
256
./cli-9749-4.json
256
./cli-9749-5.json
256
./cli-9749-10.json
256
./cli-9749-9.json
256
./cli-9749-6.json
256
./cli-9749-7.json
256
./cli-9749-1.json
256


andyfeller@Andys-MBP:~ $ for i in {2..10}; do \
    diff <(jq 'sort' cli-9749-1.json) <(jq 'sort' "cli-9749-$i.json");
done
andyfeller@Andys-MBP:~ $
```

I don't believe `gh search` caches results of these calls, however it should be easy enough to add `gh config clear-cache` between `gh search prs` calls to ensure it clears any potential cache.

### Next steps

@wknapik : I'd like you to attempt reproducing this problem while preserving the results from the `gh search prs` commands and sharing your thoughts on where you are seeing variants.
author_association: CONTRIBUTOR
user: login: wknapik
created_at: 2024-10-16T15:02:59Z
body: Hi @andyfeller. Your code does not appear to check for duplicates, only for consistency. When I run it, I get the same results you do. There's 256 results, including duplicates, identical every time, not affected by running `gh config clear-cache`.
author_association: NONE
user: login: wknapik
created_at: 2024-10-16T15:10:59Z
body: I think we might be misunderstanding each other.

The point is that there are duplicate results. There shouldn't be any, regardless of what value is passed to `--limit`.

The commands I shared are meant to illustrate that for the `--limit` value of 256, I got 56 duplicates, for 275 I got 50 duplicates and for 300 there were no duplicates.
author_association: NONE
user: login: wknapik
created_at: 2024-10-16T15:15:36Z
body: It's also interesting to note that up to a point, the duplicates are added until the total number of results returned is equal to the value of `--limit`. Past that point, there are no duplicates anymore, regardless of the `--limit` value.
author_association: NONE
user: login: wknapik
created_at: 2024-10-16T23:29:06Z
body: cli-9749-1.json ends with

```
{"number":25808},{"number":25806},{"number":25805},{"number":25804},{"number":25802},{"number":25799},{"number":25798},{"number":25797},{"number":25796},{"number":25795},{"number":25794},{"number":25793},{"number":25792},{"number":25788},{"number":25785},{"number":25784},{"number":25783},{"number":25780},{"number":25779},{"number":25777},{"number":25776},{"number":25775},{"number":25774},{"number":25771},{"number":25770},{"number":25769},{"number":25768},{"number":25767},{"number":25764},{"number":25761},{"number":25760},{"number":25759},{"number":25758},{"number":25757},{"number":25752},{"number":25750},{"number":25749},{"number":25748},{"number":25747},{"number":25744},{"number":25743},{"number":25740},{"number":25738},{"number":25737},{"number":25736},{"number":25735},{"number":25734},{"number":25731},{"number":25730},{"number":25726},{"number":25721},{"number":25720},{"number":25719},{"number":25718},{"number":25717},{"number":25716}
```

which is a repetition of an identical sequence earlier in the file - those are the 56 duplicates
author_association: NONE
user: login: wknapik
created_at: 2024-10-22T13:40:25Z
body: @andyfeller are you, or anyone else looking into this? Please at least remove the `needs-triage`/`needs-user-input` labels.
author_association: NONE
user: login: andyfeller
created_at: 2024-10-23T14:41:55Z
body: > I think we might be misunderstanding each other.
> 
> The point is that there are duplicate results. There shouldn't be any, regardless of what value is passed to `--limit`.
> 
> The commands I shared are meant to illustrate that for the `--limit` value of 256, I got 56 duplicates, for 275 I got 50 duplicates and for 300 there were no duplicates.

@wknapik : I don't think we're misunderstanding each other, but rather there isn't enough information to understand why your results are inconsistent.

> ```shell
> % gh -R brave/brave-core search prs --limit 256 --merged --merged-at ">$(date -I -d '1 month ago')" --base=master --json number -q '.[].number'|wc -l
> 256
> % gh -R brave/brave-core search prs --limit 256 --merged --merged-at ">$(date -I -d '1 month ago')" --base=master --json number -q '.[].number'|sort -u|wc -l
> 200
> ```

From these commands, we're solely counting the number of lines in a JSON document.  I don't know if the content from these calls can be shared, but one thing that is misleading to me is uniquely sorting a JSON document and counting lines vs total lines especially when there is extra lines due to JSON prettifying.



> cli-9749-1.json ends with
> 
> ```
> {"number":25808},{"number":25806},{"number":25805},{"number":25804},{"number":25802},{"number":25799},{"number":25798},{"number":25797},{"number":25796},{"number":25795},{"number":25794},{"number":25793},{"number":25792},{"number":25788},{"number":25785},{"number":25784},{"number":25783},{"number":25780},{"number":25779},{"number":25777},{"number":25776},{"number":25775},{"number":25774},{"number":25771},{"number":25770},{"number":25769},{"number":25768},{"number":25767},{"number":25764},{"number":25761},{"number":25760},{"number":25759},{"number":25758},{"number":25757},{"number":25752},{"number":25750},{"number":25749},{"number":25748},{"number":25747},{"number":25744},{"number":25743},{"number":25740},{"number":25738},{"number":25737},{"number":25736},{"number":25735},{"number":25734},{"number":25731},{"number":25730},{"number":25726},{"number":25721},{"number":25720},{"number":25719},{"number":25718},{"number":25717},{"number":25716}
> ```
> 
> which is a repetition of an identical sequence earlier in the file - those are the 56 duplicates

This is helpful, thank you!

**Could I trouble you to redirect the output of the calls to files and attach them to this issue please?**  Also running these commands with `GH_DEBUG=api` would help provide more information on what the API is returning.

While you do that, we can begin digging into the code to understand why there might be duplicates.
author_association: CONTRIBUTOR
user: login: andyfeller
created_at: 2024-10-23T14:43:21Z
body: > [@andyfeller](https://github.com/andyfeller) are you, or anyone else looking into this? Please at least remove the `needs-triage`/`needs-user-input` labels.

@wknapik : I'd like to get the actual JSON output from the commands you are running and attaching them to this issue and identify where the bug actually is before we can confirm it and figure out a path forward.

### What is `gh search prs` doing under the covers and how might it be returning duplicate results?

Doing some cursory digging, I wanted to highlight the core of PR searching code:

https://github.com/cli/cli/blob/e390874fef5eb5545db2c4ff0861cb1c3e83dd6f/pkg/search/searcher.go#L133-L195

- initial call to the REST search API retrieves as much as we can get
- it iterates onto next pages as the API response indicates there is more data
- we loop as there are pages and we still need data

Again, I think having the actual output from the commands before running them through `wc -l` and the `GH_DEBUG=api` would let us understand whether there are responses coming back from the GitHub API that are retrieving more data than necessary or if the API is returning redundant results. 🤔 
author_association: CONTRIBUTOR
user: login: wknapik
created_at: 2024-10-24T14:15:42Z
body: > From these commands, we're solely counting the number of lines in a JSON document. I don't know if the content from these calls can be shared, but one thing that is misleading to me is uniquely sorting a JSON document and counting lines vs total lines especially when there is extra lines due to JSON prettifying.

This isn't a JSON document. It's a list of integers (PR numbers), one per line.

Here's the original file produced by your code snippet on 2024-10-16:
```bash
gh search prs --limit 256 --merged --merged-at ">2024-09-14" --base master  --repo brave/brave-core --json number > "cli-9749-$i.json";
```
[cli-9749-1.json](https://github.com/user-attachments/files/17507720/cli-9749-1.json)

> Could I trouble you to redirect the output of the calls to files and attach them to this issue please? Also running these commands with GH_DEBUG=api would help provide more information on what the API is returning.

Here's a file produced just now via:
```bash
GH_DEBUG=api gh search prs --limit 256 --merged --merged-at ">2024-09-14" --base master  --repo brave/brave-core --json number >cli-debug.txt 2>&1
```
[cli-debug.txt](https://github.com/user-attachments/files/17507862/cli-debug.txt)

The contents of the file is now different than a week ago, but it still contains 56 duplicates:
```console
% jq -r '.[].number'|wc -l      
256
% jq -r '.[].number'|sort -u|wc -l
200
%
```

Once again, the duplicates are at the end - the following sequence is a repetition of an identical sequence of 56 numbers earlier in the file:
```json
{"number":25926},{"number":25923},{"number":25920},{"number":25918},{"number":25917},{"number":25914},{"number":25913},{"number":25912},{"number":25911},{"number":25910},{"number":25909},{"number":25908},{"number":25907},{"number":25906},{"number":25905},{"number":25904},{"number":25898},{"number":25897},{"number":25896},{"number":25895},{"number":25892},{"number":25889},{"number":25888},{"number":25887},{"number":25883},{"number":25880},{"number":25879},{"number":25878},{"number":25877},{"number":25875},{"number":25874},{"number":25873},{"number":25870},{"number":25868},{"number":25865},{"number":25864},{"number":25863},{"number":25862},{"number":25861},{"number":25860},{"number":25857},{"number":25855},{"number":25854},{"number":25853},{"number":25852},{"number":25851},{"number":25850},{"number":25847},{"number":25846},{"number":25845},{"number":25844},{"number":25843},{"number":25842},{"number":25840},{"number":25839},{"number":25838}
```
author_association: NONE
user: login: andyfeller
created_at: 2024-10-31T20:02:59Z
body: Thanks, @wknapik, I see what you mean and trying to reverse engineer how this is happening 👍 

<details>
<summary>
  <b>Personal investigation notes</b>
</summary>
<p>

From the following commands, it appears that `gh search prs` is issuing the appropriate GitHub Search API calls to pull data for `page` and `per_page` 🤔 

```shell
GH_DEBUG=api gh search prs --limit 256 --merged --merged-at ">2024-09-14" --base master  --repo brave/brave-core --json number
```

- `GET /search/issues?page=1&per_page=100&q=base%3Amaster+is%3Amerged+merged%3A%3E2024-09-14+repo%3Abrave%2Fbrave-core+type%3Apr HTTP/1.1`
- `GET /search/issues?page=2&per_page=100&q=base%3Amaster+is%3Amerged+merged%3A%3E2024-09-14+repo%3Abrave%2Fbrave-core+type%3Apr HTTP/1.1`
- `GET /search/issues?page=3&per_page=56&q=base%3Amaster+is%3Amerged+merged%3A%3E2024-09-14+repo%3Abrave%2Fbrave-core+type%3Apr HTTP/1.1`

```shell
GH_DEBUG=api gh search prs --limit 275 --merged --merged-at ">2024-09-14" --base master  --repo brave/brave-core --json number
```

- `GET /search/issues?page=1&per_page=100&q=base%3Amaster+is%3Amerged+merged%3A%3E2024-09-14+repo%3Abrave%2Fbrave-core+type%3Apr HTTP/1.1`
- `GET /search/issues?page=2&per_page=100&q=base%3Amaster+is%3Amerged+merged%3A%3E2024-09-14+repo%3Abrave%2Fbrave-core+type%3Apr HTTP/1.1`
- `GET /search/issues?page=3&per_page=75&q=base%3Amaster+is%3Amerged+merged%3A%3E2024-09-14+repo%3Abrave%2Fbrave-core+type%3Apr HTTP/1.1`

Yet as reported, it seems there redundant search results are being returned.

Dropping the `--json number` and processing the displayed table, you can confirm the duplicates so it isn't something unique to `--json`:

> ```shell
> brave/brave-core  #25905  Don't store ContentSettingsClient ref as it may be destroryed.                                                   CI/skip-ios                                                                                                                                      about 21 days ago
> brave/brave-core  #25905  Don't store ContentSettingsClient ref as it may be destroryed.                                                   CI/skip-ios                                                                                                                                      about 21 days ago
> brave/brave-core  #25906  [ubsan] Fix null deref on `OnboardingTest`                                                                                                                                                                                                                        about 21 days ago
> brave/brave-core  #25906  [ubsan] Fix null deref on `OnboardingTest`                                                                                                                                                                                                                        about 21 days ago
> brave/brave-core  #25907  Move Inactive tabs setting to Appearance section                                                                 CI/storybook-url                                                                                                                                 about 17 days ago
> brave/brave-core  #25907  Move Inactive tabs setting to Appearance section                                                                 CI/storybook-url                                                                                                                                 about 17 days ago
> brave/brave-core  #25908  [ads] Add missing brave_adaptive_captcha dependency                                                                                                                                                                                                               about 22 days ago
> brave/brave-core  #25908  [ads] Add missing brave_adaptive_captcha dependency                                                                                                                                                                                                               about 22 days ago
> brave/brave-core  #25909  fix(privacy: Shred button spacing in private tabs                                                                CI/skip-android, CI/skip-macos-x64, CI/skip-windows-x64, CI/skip-macos-arm64                                                                     about 16 days ago
> brave/brave-core  #25909  fix(privacy: Shred button spacing in private tabs                                                                CI/skip-android, CI/skip-macos-x64, CI/skip-windows-x64, CI/skip-macos-arm64                                                                     about 16 days ago
> brave/brave-core  #25910  Branch migration - master branch                                                                                 CI/skip, CI/run-audit-deps                                                                                                                       about 21 days ago
> brave/brave-core  #25910  Branch migration - master branch                                                                                 CI/skip, CI/run-audit-deps                                                                                                                       about 21 days ago
> brave/brave-core  #25911  remove include rules that no longer apply                                                                        CI/run-audit-deps, feature/web3/wallet, feature/web3/wallet/core, puLL-Merge                                                                     about 20 days ago
> brave/brave-core  #25911  remove include rules that no longer apply                                                                        CI/run-audit-deps, feature/web3/wallet, feature/web3/wallet/core, puLL-Merge                                                                     about 20 days ago
> brave/brave-core  #25912  Hide caption options in media control popup                                                                                                                                                                                                                       about 21 days ago
> brave/brave-core  #25912  Hide caption options in media control popup                                                                                                                                                                                                                       about 21 days ago
> brave/brave-core  #25913  [Vertical Tabs]: Add hover effect                                                                                                                                                                                                                                 about 14 days ago
> brave/brave-core  #25913  [Vertical Tabs]: Add hover effect                                                                                                                                                                                                                                 about 14 days ago
> brave/brave-core  #25914  [Brave News]: Don't use TestingProfile in component unittests                                                    puLL-Merge                                                                                                                                       about 15 days ago
> brave/brave-core  #25914  [Brave News]: Don't use TestingProfile in component unittests                                                    puLL-Merge                                                                                                                                       about 15 days ago
> brave/brave-core  #25917  [ubsan] Avoid early downcast of `BraveTabStrip`                                                                                                                                                                                                                   about 21 days ago
> brave/brave-core  #25917  [ubsan] Avoid early downcast of `BraveTabStrip`                                                                                                                                                                                                                   about 21 days ago
> brave/brave-core  #25918  [ubsan] `SidebarContainerView` requires precocious downcast                                                                                                                                                                                                       about 21 days ago
> brave/brave-core  #25918  [ubsan] `SidebarContainerView` requires precocious downcast                                                                                                                                                                                                       about 21 days ago
> brave/brave-core  #25920  Replace requests to `laptop-updates.brave.com` with requests to `usage-ping.brave.com` and `feedback.brave.com`  CI/run-network-audit                                                                                                                             about 10 days ago
> brave/brave-core  #25920  Replace requests to `laptop-updates.brave.com` with requests to `usage-ping.brave.com` and `feedback.brave.com`  CI/run-network-audit                                                                                                                             about 10 days ago
> brave/brave-core  #25923  Fixed navigation bar color at private tab                                                                                                                                                                                                                         about 19 days ago
> brave/brave-core  #25923  Fixed navigation bar color at private tab                                                                                                                                                                                                                         about 19 days ago
> brave/brave-core  #25926  [CodeHealth] Avoid unnecessary uses of `vector`                                                                                                                                                                                                                   about 20 days ago
> brave/brave-core  #25926  [CodeHealth] Avoid unnecessary uses of `vector`                                                                                                                                                                                                                   about 20 days ago
> brave/brave-core  #25927  Updated Tor button style.                                                                                                                                                                                                                                         about 9 days ago
> brave/brave-core  #25927  Updated Tor button style.                                                                                                                                                                                                                                         about 9 days ago
> brave/brave-core  #25928  [CodeHealth] Remove empty param lists for lambdas                                                                feature/web3/wallet, feature/web3/wallet/core                                                                                                    about 20 days ago
> brave/brave-core  #25928  [CodeHealth] Remove empty param lists for lambdas                                                                feature/web3/wallet, feature/web3/wallet/core                                                                                                    about 20 days ago
> brave/brave-core  #25929  Leo annual subscription on Android                                                                               CI/skip-macos-x64, CI/skip-ios, CI/skip-windows-x64, puLL-Merge, CI/skip-macos-arm64                                                             about 13 days ago
> brave/brave-core  #25929  Leo annual subscription on Android                                                                               CI/skip-macos-x64, CI/skip-ios, CI/skip-windows-x64, puLL-Merge, CI/skip-macos-arm64                                                             about 13 days ago
> brave/brave-core  #25931  Internationalization Support for Leo                                                                             CI/storybook-url, puLL-Merge                                                                                                                     about 3 days ago
> brave/brave-core  #25931  Internationalization Support for Leo                                                                             CI/storybook-url, puLL-Merge                                                                                                                     about 3 days ago
> brave/brave-core  #25934  Upgrade from Chromium 130.0.6723.31 to Chromium 130.0.6723.44                                                    CI/run-network-audit, CI/run-audit-deps                                                                                                          about 20 days ago
> brave/brave-core  #25934  Upgrade from Chromium 130.0.6723.31 to Chromium 130.0.6723.44                                                    CI/run-network-audit, CI/run-audit-deps                                                                                                          about 20 days ago
> brave/brave-core  #25935  Add secteam reviews for SCT exceptions                                                                           CI/skip                                                                                                                                          about 20 days ago
> brave/brave-core  #25935  Add secteam reviews for SCT exceptions                                                                           CI/skip                                                                                                                                          about 20 days ago
> brave/brave-core  #25936  [CrButton]: Add custom size support                                                                              CI/storybook-url                                                                                                                                 about 20 days ago
> brave/brave-core  #25936  [CrButton]: Add custom size support                                                                              CI/storybook-url                                                                                                                                 about 20 days ago
> brave/brave-core  #25937  Change default font in desktop settings to system fonts                                                          CI/storybook-url                                                                                                                                 about 16 days ago
> brave/brave-core  #25937  Change default font in desktop settings to system fonts                                                          CI/storybook-url                                                                                                                                 about 16 days ago
> brave/brave-core  #25938  Update rate dialog UI                                                                                            CI/skip-macos-x64, CI/skip-ios, CI/skip-windows-x64, unused-CI/skip-linux-x64, CI/skip-macos-arm64                                               about 13 days ago
> brave/brave-core  #25938  Update rate dialog UI                                                                                            CI/skip-macos-x64, CI/skip-ios, CI/skip-windows-x64, unused-CI/skip-linux-x64, CI/skip-macos-arm64                                               about 13 days ago
> brave/brave-core  #25939  Update laptop icon for sync                                                                                      CI/skip-android, CI/skip-macos-x64, CI/skip-windows-x64, unused-CI/skip-linux-x64, CI/skip-macos-arm64                                           about 20 days ago
> brave/brave-core  #25939  Update laptop icon for sync                                                                                      CI/skip-android, CI/skip-macos-x64, CI/skip-windows-x64, unused-CI/skip-linux-x64, CI/skip-macos-arm64                                           about 20 days ago
> brave/brave-core  #25940  [ubsan] Fix `BraveNewsController*` null deref                                                                                                                                                                                                                     about 20 days ago
> brave/brave-core  #25940  [ubsan] Fix `BraveNewsController*` null deref                                                                                                                                                                                                                     about 20 days ago
> brave/brave-core  #25943  [DanglingPtr] Fix `AdBlockPrefService` dangling access                                                                                                                                                                                                            about 20 days ago
> brave/brave-core  #25943  [DanglingPtr] Fix `AdBlockPrefService` dangling access                                                                                                                                                                                                            about 20 days ago
> brave/brave-core  #25944  [DanglingPtr] Add observation to `ViewCounterService`                                                                                                                                                                                                             about 20 days ago
> brave/brave-core  #25944  [DanglingPtr] Add observation to `ViewCounterService`                                                                                                                                                                                                             about 20 days ago
> brave/brave-core  #25946  [DanglinPtr] Fix various unit test violations                                                                                                                                                                                                                     about 20 days ago
> brave/brave-core  #25946  [DanglinPtr] Fix various unit test violations                                                                                                                                                                                                                     about 20 days ago
> brave/brave-core  #25947  Fixed flaky sidebar browser test                                                                                                                                                                                                                                  about 20 days ago
> brave/brave-core  #25947  Fixed flaky sidebar browser test                                                                                                                                                                                                                                  about 20 days ago
> brave/brave-core  #25948  Adjust create wallet account ui                                                                                  CI/storybook-url, feature/web3/wallet, feature/web3/wallet/core, puLL-Merge                                                                      about 10 days ago
> brave/brave-core  #25948  Adjust create wallet account ui                                                                                  CI/storybook-url, feature/web3/wallet, feature/web3/wallet/core, puLL-Merge                                                                      about 10 days ago
> brave/brave-core  #25949  Remove Greaselion dependency from Rewards service (v2)                                                           CI/storybook-url, CI/run-upstream-tests, needs-security-review, puLL-Merge                                                                       about 6 days ago
> brave/brave-core  #25949  Remove Greaselion dependency from Rewards service (v2)                                                           CI/storybook-url, CI/run-upstream-tests, needs-security-review, puLL-Merge                                                                       about 6 days ago
> brave/brave-core  #25952  Fix placement of inactive tabs setting                                                                                                                                                                                                                            about 17 days ago
> brave/brave-core  #25952  Fix placement of inactive tabs setting                                                                                                                                                                                                                            about 17 days ago
> brave/brave-core  #25953  fix(wallet): Old Activity Route Persisted in Panel                                                               CI/storybook-url, feature/web3/wallet                                                                                                            about 16 days ago
> brave/brave-core  #25953  fix(wallet): Old Activity Route Persisted in Panel                                                               CI/storybook-url, feature/web3/wallet                                                                                                            about 16 days ago
> brave/brave-core  #25954  fix(privacy): Fix Shred on app close                                                                             CI/skip-android, CI/skip-macos-x64, CI/skip-windows-x64, CI/skip-macos-arm64                                                                     about 10 days ago
> brave/brave-core  #25954  fix(privacy): Fix Shred on app close                                                                             CI/skip-android, CI/skip-macos-x64, CI/skip-windows-x64, CI/skip-macos-arm64                                                                     about 10 days ago
> brave/brave-core  #25955  [Rewards 3.0] Fix Default AC amount display                                                                      CI/storybook-url                                                                                                                                 about 17 days ago
> brave/brave-core  #25955  [Rewards 3.0] Fix Default AC amount display                                                                      CI/storybook-url                                                                                                                                 about 17 days ago
> brave/brave-core  #25956  Content picker follow up                                                                                         CI/storybook-url                                                                                                                                 about 16 days ago
> brave/brave-core  #25956  Content picker follow up                                                                                         CI/storybook-url                                                                                                                                 about 16 days ago
> brave/brave-core  #25957  fix(wallet): Fix tx submission screen gets dismissed automatically after tx is confirmed                         CI/skip-android, CI/skip-macos-x64, CI/skip-windows-x64, CI/skip-macos-arm64, CI/skip-teamcity                                                   about 16 days ago
> brave/brave-core  #25957  fix(wallet): Fix tx submission screen gets dismissed automatically after tx is confirmed                         CI/skip-android, CI/skip-macos-x64, CI/skip-windows-x64, CI/skip-macos-arm64, CI/skip-teamcity                                                   about 16 days ago
> brave/brave-core  #25960  Improves teardown to prevent failing test on Win                                                                 tests, release-notes/exclude, QA/No                                                                                                              about 13 days ago
> brave/brave-core  #25960  Improves teardown to prevent failing test on Win                                                                 tests, release-notes/exclude, QA/No                                                                                                              about 13 days ago
> brave/brave-core  #25961  [CodeHealth][ads] Use `base::ScopedObservation` pt.1                                                                                                                                                                                                              about 17 days ago
> brave/brave-core  #25961  [CodeHealth][ads] Use `base::ScopedObservation` pt.1                                                                                                                                                                                                              about 17 days ago
> brave/brave-core  #25962  [CodeHealth] Use `string_view` with l10n utils                                                                                                                                                                                                                    about 17 days ago
> brave/brave-core  #25962  [CodeHealth] Use `string_view` with l10n utils                                                                                                                                                                                                                    about 17 days ago
> brave/brave-core  #25963  Roll perf profiles update (update-perf-profile-2024-10-13-16-16)                                                 CI/skip                                                                                                                                          about 17 days ago
> brave/brave-core  #25963  Roll perf profiles update (update-perf-profile-2024-10-13-16-16)                                                 CI/skip                                                                                                                                          about 17 days ago
> brave/brave-core  #25964  update adblock-rust to v0.9.2                                                                                    CI/run-audit-deps                                                                                                                                about 10 days ago
> brave/brave-core  #25964  update adblock-rust to v0.9.2                                                                                    CI/run-audit-deps                                                                                                                                about 10 days ago
> brave/brave-core  #25966  [Toggle]: Fix toggles not toggling                                                                               CI/storybook-url                                                                                                                                 about 20 hours ago
> brave/brave-core  #25966  [Toggle]: Fix toggles not toggling                                                                               CI/storybook-url                                                                                                                                 about 20 hours ago
> brave/brave-core  #25971  Fixed device duplication when Google Account cookies are deleted                                                                                                                                                                                                  about 13 days ago
> brave/brave-core  #25971  Fixed device duplication when Google Account cookies are deleted                                                                                                                                                                                                  about 13 days ago
> brave/brave-core  #25972  [CodeHealth][rewards] Use `base::ScopedObservation` pt.2                                                         puLL-Merge                                                                                                                                       about 16 days ago
> brave/brave-core  #25972  [CodeHealth][rewards] Use `base::ScopedObservation` pt.2                                                         puLL-Merge                                                                                                                                       about 16 days ago
> brave/brave-core  #25975  encapsulate rust json code in cpp                                                                                feature/web3/wallet, feature/web3/wallet/core, puLL-Merge                                                                                        about 15 days ago
> brave/brave-core  #25975  encapsulate rust json code in cpp                                                                                feature/web3/wallet, feature/web3/wallet/core, puLL-Merge                                                                                        about 15 days ago
> brave/brave-core  #25977  [ads] Deprecate `ReplaceBraveSchemeWithChromeScheme`                                                                                                                                                                                                              about 16 days ago
> brave/brave-core  #25977  [ads] Deprecate `ReplaceBraveSchemeWithChromeScheme`                                                                                                                                                                                                              about 16 days ago
> brave/brave-core  #25978  [CodeHealth] Use span of bytes to read arbitrary data                                                            puLL-Merge                                                                                                                                       about 14 days ago
> brave/brave-core  #25978  [CodeHealth] Use span of bytes to read arbitrary data                                                            puLL-Merge                                                                                                                                       about 14 days ago
> brave/brave-core  #25979  Add IPFS Component build flag, some other review suggestions follow up                                                                                                                                                                                            about 14 days ago
> brave/brave-core  #25979  Add IPFS Component build flag, some other review suggestions follow up                                                                                                                                                                                            about 14 days ago
> ```

</p>
</details> 

Following up internally with issue and PR teams to understand why changing the `per_page` on search API calls is resulting in results from the previous page being returned again.

In capturing `GH_API=debug` to see the actual API endpoints called, I can confirm GitHub API is doing something unexpected:

```shell
gh api "/search/issues?page=1&per_page=100&q=base%3Amaster+is%3Amerged+merged%3A%3E2024-09-14+repo%3Abrave%2Fbrave-core+type%3Apr" > 256-part-1.json
gh api "/search/issues?page=2&per_page=100&q=base%3Amaster+is%3Amerged+merged%3A%3E2024-09-14+repo%3Abrave%2Fbrave-core+type%3Apr" > 256-part-2.json
gh api "/search/issues?page=3&per_page=56&q=base%3Amaster+is%3Amerged+merged%3A%3E2024-09-14+repo%3Abrave%2Fbrave-core+type%3Apr" > 256-part-3.json

gh api "/search/issues?page=1&per_page=100&q=base%3Amaster+is%3Amerged+merged%3A%3E2024-09-14+repo%3Abrave%2Fbrave-core+type%3Apr" > 275-part-1.json
gh api "/search/issues?page=2&per_page=100&q=base%3Amaster+is%3Amerged+merged%3A%3E2024-09-14+repo%3Abrave%2Fbrave-core+type%3Apr" > 275-part-2.json
gh api "/search/issues?page=3&per_page=75&q=base%3Amaster+is%3Amerged+merged%3A%3E2024-09-14+repo%3Abrave%2Fbrave-core+type%3Apr" > 275-part-3.json
```

> [!NOTE]
> It appears `gh search` results are unique when `--limit` is a denomination of 100.
author_association: CONTRIBUTOR
user: login: andyfeller
created_at: 2024-11-01T12:09:25Z
body: @wknapik : I've been able to confirm that the PR Search API is apparently optimized for `--limit` in 100 intervals.  Unfortunately, that leaves `gh` in a weird situation:

1. Does `gh` attempt to add intelligence to de-duplicate results and continue pulling more data than specified to try filling up the bucket?

2. Do `gh` users specify `--limit` < 100 or in intervals of 100 until the search API indexing can be properly fixed?

I'm hesitant on the former, however I'm going to bring this up with the other maintainers.


_UPDATED: Able to confirm this affects issues in `gh search issues --limit 256 --repo cli/cli | sort | uniq -d | wc -l`_
author_association: CONTRIBUTOR
user: login: wknapik
created_at: 2024-11-01T14:26:14Z
body: A server-side optimization would be transparent to the user.
If a valid request returns and invalid result, I'd call that a bug.

I don't believe documenting a bug makes it not a bug.
And shifting the burden of working around a bug to the end user is arguably user-hostile.

This is a public API. The CLI is not its only consumer, so while it could try to hide the issue, to the benefit of the user, ultimately, the problem should be resolved at the source.

That's just my 2c. Am I misunderstanding something?
author_association: NONE
user: login: andyfeller
created_at: 2024-11-01T15:08:48Z
body: > A server-side optimization would be transparent to the user. If a valid request returns and invalid result, I'd call that a bug.

So let me rephrase an earlier statement: the [GitHub Search REST APIs](https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28) aren't the problem here so much as how `gh search` commands are trying to use it.

These endpoints don't have a concept of cursor like GraphQL, so how `gh search` is trying to retrieve partial buckets is the error here as the partial buckets are the wrong coordinates:

https://github.com/cli/cli/blob/30066b0042d0c5928d959e288144300cb28196c9/pkg/search/searcher.go#L61-L155

As such, this appears to be an issue with all of the `gh search` commands, so something for us to fix 👍 

Thanks to @williammartin for throwing together a proof of concept on how what fixing these commands would entail in https://github.com/cli/cli/compare/trunk...wm/spike-search-dupe-fix; again I think this affects the whole commandset and would need thorough testing:

https://github.com/cli/cli/blob/2c9ed1f2a5b35dec636a3c5b53447d55ef44ae07/pkg/search/searcher.go#L133-L159
author_association: CONTRIBUTOR
user: login: wknapik
created_at: 2024-11-01T15:46:17Z
body: I see, thanks for the explanation
author_association: NONE
user: login: williammartin
created_at: 2024-11-12T11:42:24Z
body: ## Proposed Acceptance Criteria

**Given** I have more than 100 results for a search query
**When** I run `gh search <resource> --limit 110`
**Then** I get 110 unique items (rather than having the last 10 items be duplicated)

@andyfeller does that about cover the A/C for this?
author_association: MEMBER
user: login: bodgit
created_at: 2025-03-31T08:51:20Z
body: This has happened to me every so often when the results tip over 100. Using 200 as the limit works much better so I'll do that until this hopefully gets fixed
author_association: NONE

Diff:
diff --git a/pkg/search/result.go b/pkg/search/result.go
index 0c7c43cd7c5..0b9d1ab168e 100644
--- a/pkg/search/result.go
+++ b/pkg/search/result.go
@@ -93,25 +93,29 @@ var PullRequestFields = append(IssueFields,
 type CodeResult struct {
 	IncompleteResults bool   `json:"incomplete_results"`
 	Items             []Code `json:"items"`
-	Total             int    `json:"total_count"`
+	// Number of code search results matching the query on the server. Ignoring limit.
+	Total int `json:"total_count"`
 }
 
 type CommitsResult struct {
 	IncompleteResults bool     `json:"incomplete_results"`
 	Items             []Commit `json:"items"`
-	Total             int      `json:"total_count"`
+	// Number of commits matching the query on the server. Ignoring limit.
+	Total int `json:"total_count"`
 }
 
 type RepositoriesResult struct {
 	IncompleteResults bool         `json:"incomplete_results"`
 	Items             []Repository `json:"items"`
-	Total             int          `json:"total_count"`
+	// Number of repositories matching the query on the server. Ignoring limit.
+	Total int `json:"total_count"`
 }
 
 type IssuesResult struct {
 	IncompleteResults bool    `json:"incomplete_results"`
 	Items             []Issue `json:"items"`
-	Total             int     `json:"total_count"`
+	// Number of isssues matching the query on the server. Ignoring limit.
+	Total int `json:"total_count"`
 }
 
 type Code struct {
diff --git a/pkg/search/searcher.go b/pkg/search/searcher.go
index 4168dc7f3a5..7cbd355623b 100644
--- a/pkg/search/searcher.go
+++ b/pkg/search/searcher.go
@@ -14,6 +14,7 @@ import (
 )
 
 const (
+	// GitHub API has a limit of 100 per page
 	maxPerPage = 100
 	orderKey   = "order"
 	sortKey    = "sort"
@@ -60,100 +61,145 @@ func NewSearcher(client *http.Client, host string) Searcher {
 
 func (s searcher) Code(query Query) (CodeResult, error) {
 	result := CodeResult{}
-	toRetrieve := query.Limit
+
 	var resp *http.Response
 	var err error
-	for toRetrieve > 0 {
-		query.Limit = min(toRetrieve, maxPerPage)
+
+	// We will request either the query limit if it's less than 1 page, or our max page size.
+	// This number doesn't change to keep a valid offset.
+	//
+	// For example, say we want 150 items out of 500.
+	// We request page #1 for 100 items and get items 0 to 99.
+	// Then we request page #2 for 100 items, we get items 100 to 199 and only keep 100 to 149.
+	// If we were to request page #2 for 50 items, we would instead get items 50 to 99.
+	numItemsToRetrieve := query.Limit
+	query.Limit = min(numItemsToRetrieve, maxPerPage)
+
+	for numItemsToRetrieve > 0 {
 		query.Page = nextPage(resp)
 		if query.Page == 0 {
 			break
 		}
+
 		page := CodeResult{}
 		resp, err = s.search(query, &page)
 		if err != nil {
 			return result, err
 		}
+
+		// If we're going to reach the requested limit, only add that many items,
+		// otherwise add all the results.
+		numItemsToAdd := min(len(page.Items), numItemsToRetrieve)
 		result.IncompleteResults = page.IncompleteResults
+		// The API returns how many items match the query in every response.
+		// With the example above, this would be 500.
 		result.Total = page.Total
-		result.Items = append(result.Items, page.Items...)
-		toRetrieve = toRetrieve - len(page.Items)
+		result.Items = append(result.Items, page.Items[:numItemsToAdd]...)
+		numItemsToRetrieve = numItemsToRetrieve - numItemsToAdd
 	}
+
 	return result, nil
 }
 
 func (s searcher) Commits(query Query) (CommitsResult, error) {
 	result := CommitsResult{}
-	toRetrieve := query.Limit
+
 	var resp *http.Response
 	var err error
-	for toRetrieve > 0 {
-		query.Limit = min(toRetrieve, maxPerPage)
+
+	numItemsToRetrieve := query.Limit
+	query.Limit = min(numItemsToRetrieve, maxPerPage)
+
+	for numItemsToRetrieve > 0 {
 		query.Page = nextPage(resp)
 		if query.Page == 0 {
 			break
 		}
+
 		page := CommitsResult{}
 		resp, err = s.search(query, &page)
 		if err != nil {
 			return result, err
 		}
+
+		numItemsToAdd := min(len(page.Items), numItemsToRetrieve)
 		result.IncompleteResults = page.IncompleteResults
 		result.Total = page.Total
-		result.Items = append(result.Items, page.Items...)
-		toRetrieve = toRetrieve - len(page.Items)
+		result.Items = append(result.Items, page.Items[:numItemsToAdd]...)
+		numItemsToRetrieve = numItemsToRetrieve - numItemsToAdd
 	}
 	return result, nil
 }
 
 func (s searcher) Repositories(query Query) (RepositoriesResult, error) {
 	result := RepositoriesResult{}
-	toRetrieve := query.Limit
+
 	var resp *http.Response
 	var err error
-	for toRetrieve > 0 {
-		query.Limit = min(toRetrieve, maxPerPage)
+
+	numItemsToRetrieve := query.Limit
+	query.Limit = min(numItemsToRetrieve, maxPerPage)
+
+	for numItemsToRetrieve > 0 {
 		query.Page = nextPage(resp)
 		if query.Page == 0 {
 			break
 		}
+
 		page := RepositoriesResult{}
 		resp, err = s.search(query, &page)
 		if err != nil {
 			return result, err
 		}
+
+		numItemsToAdd := min(len(page.Items), numItemsToRetrieve)
 		result.IncompleteResults = page.IncompleteResults
 		result.Total = page.Total
-		result.Items = append(result.Items, page.Items...)
-		toRetrieve = toRetrieve - len(page.Items)
+		result.Items = append(result.Items, page.Items[:numItemsToAdd]...)
+		numItemsToRetrieve = numItemsToRetrieve - numItemsToAdd
 	}
 	return result, nil
 }
 
 func (s searcher) Issues(query Query) (IssuesResult, error) {
 	result := IssuesResult{}
-	toRetrieve := query.Limit
+
 	var resp *http.Response
 	var err error
-	for toRetrieve > 0 {
-		query.Limit = min(toRetrieve, maxPerPage)
+
+	numItemsToRetrieve := query.Limit
+	query.Limit = min(numItemsToRetrieve, maxPerPage)
+
+	for numItemsToRetrieve > 0 {
 		query.Page = nextPage(resp)
 		if query.Page == 0 {
 			break
 		}
+
 		page := IssuesResult{}
 		resp, err = s.search(query, &page)
 		if err != nil {
 			return result, err
 		}
+
+		numItemsToAdd := min(len(page.Items), numItemsToRetrieve)
 		result.IncompleteResults = page.IncompleteResults
 		result.Total = page.Total
-		result.Items = append(result.Items, page.Items...)
-		toRetrieve = toRetrieve - len(page.Items)
+		result.Items = append(result.Items, page.Items[:numItemsToAdd]...)
+		numItemsToRetrieve = numItemsToRetrieve - numItemsToAdd
 	}
 	return result, nil
 }
 
+// search makes a single-page REST search request for code, commits, issues, prs, or repos.
+//
+// The result argument is populated with the following information:
+//
+// - Total: the number of search results matching the query, which may exceed the number of items returned
+// - IncompleteResults: whether the search request exceeded search time limit, potentially being incomplete
+// - Items: the actual matching search results, up to 100 max items per page
+//
+// For more information, see https://docs.github.com/en/rest/search/search?apiVersion=2022-11-28.
 func (s searcher) search(query Query, result interface{}) (*http.Response, error) {
 	path := fmt.Sprintf("%ssearch/%s", ghinstance.RESTPrefix(s.host), query.Kind)
 	qs := url.Values{}
@@ -236,10 +282,15 @@ func handleHTTPError(resp *http.Response) error {
 	return httpError
 }
 
+// https://docs.github.com/en/rest/using-the-rest-api/using-pagination-in-the-rest-api
 func nextPage(resp *http.Response) (page int) {
 	if resp == nil {
 		return 1
 	}
+
+	// When using pagination, responses get a "Link" field in their header.
+	// When a next page is available, "Link" contains a link to the next page
+	// tagged with rel="next".
 	for _, m := range linkRE.FindAllStringSubmatch(resp.Header.Get("Link"), -1) {
 		if !(len(m) > 2 && m[2] == "next") {
 			continue
diff --git a/pkg/search/searcher_test.go b/pkg/search/searcher_test.go
index 8642feed097..e893c9a3b92 100644
--- a/pkg/search/searcher_test.go
+++ b/pkg/search/searcher_test.go
@@ -1,8 +1,10 @@
 package search
 
 import (
+	"fmt"
 	"net/http"
 	"net/url"
+	"strconv"
 	"testing"
 
 	"github.com/MakeNowJust/heredoc"
@@ -46,10 +48,14 @@ func TestSearcherCode(t *testing.T) {
 			httpStubs: func(reg *httpmock.Registry) {
 				reg.Register(
 					httpmock.QueryMatcher("GET", "search/code", values),
-					httpmock.JSONResponse(CodeResult{
-						IncompleteResults: false,
-						Items:             []Code{{Name: "file.go"}},
-						Total:             1,
+					httpmock.JSONResponse(map[string]interface{}{
+						"incomplete_results": false,
+						"total_count":        1,
+						"items": []interface{}{
+							map[string]interface{}{
+								"name": "file.go",
+							},
+						},
 					}),
 				)
 			},
@@ -66,10 +72,14 @@ func TestSearcherCode(t *testing.T) {
 			httpStubs: func(reg *httpmock.Registry) {
 				reg.Register(
 					httpmock.QueryMatcher("GET", "api/v3/search/code", values),
-					httpmock.JSONResponse(CodeResult{
-						IncompleteResults: false,
-						Items:             []Code{{Name: "file.go"}},
-						Total:             1,
+					httpmock.JSONResponse(map[string]interface{}{
+						"incomplete_results": false,
+						"total_count":        1,
+						"items": []interface{}{
+							map[string]interface{}{
+								"name": "file.go",
+							},
+						},
 					}),
 				)
 			},
@@ -84,25 +94,83 @@ func TestSearcherCode(t *testing.T) {
 			},
 			httpStubs: func(reg *httpmock.Registry) {
 				firstReq := httpmock.QueryMatcher("GET", "search/code", values)
-				firstRes := httpmock.JSONResponse(CodeResult{
-					IncompleteResults: false,
-					Items:             []Code{{Name: "file.go"}},
-					Total:             2,
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        2,
+					"items": []interface{}{
+						map[string]interface{}{
+							"name": "file.go",
+						},
+					},
+				})
+				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/code?page=2&per_page=30&q=org%3Agithub>; rel="next"`)
+				secondReq := httpmock.QueryMatcher("GET", "search/code", url.Values{
+					"page":     []string{"2"},
+					"per_page": []string{"30"},
+					"q":        []string{"keyword language:go"},
+				})
+				secondRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        2,
+					"items": []interface{}{
+						map[string]interface{}{
+							"name": "file2.go",
+						},
+					},
+				})
+				reg.Register(firstReq, firstRes)
+				reg.Register(secondReq, secondRes)
+			},
+		},
+		{
+			name: "collect full and partial pages under total number of matching search results",
+			query: Query{
+				Keywords: []string{"keyword"},
+				Kind:     "code",
+				Limit:    110,
+				Qualifiers: Qualifiers{
+					Language: "go",
 				},
-				)
+			},
+			result: CodeResult{
+				IncompleteResults: false,
+				Items: initialize(0, 110, func(i int) Code {
+					return Code{
+						Name: fmt.Sprintf("name%d.go", i),
+					}
+				}),
+				Total: 287,
+			},
+			httpStubs: func(reg *httpmock.Registry) {
+				firstReq := httpmock.QueryMatcher("GET", "search/code", url.Values{
+					"page":     []string{"1"},
+					"per_page": []string{"100"},
+					"q":        []string{"keyword language:go"},
+				})
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(0, 100, func(i int) interface{} {
+						return map[string]interface{}{
+							"name": fmt.Sprintf("name%d.go", i),
+						}
+					}),
+				})
 				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/code?page=2&per_page=100&q=org%3Agithub>; rel="next"`)
 				secondReq := httpmock.QueryMatcher("GET", "search/code", url.Values{
 					"page":     []string{"2"},
-					"per_page": []string{"29"},
+					"per_page": []string{"100"},
 					"q":        []string{"keyword language:go"},
-				},
-				)
-				secondRes := httpmock.JSONResponse(CodeResult{
-					IncompleteResults: false,
-					Items:             []Code{{Name: "file2.go"}},
-					Total:             2,
-				},
-				)
+				})
+				secondRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(100, 200, func(i int) interface{} {
+						return map[string]interface{}{
+							"name": fmt.Sprintf("name%d.go", i),
+						}
+					}),
+				})
 				reg.Register(firstReq, firstRes)
 				reg.Register(secondReq, secondRes)
 			},
@@ -201,10 +269,14 @@ func TestSearcherCommits(t *testing.T) {
 			httpStubs: func(reg *httpmock.Registry) {
 				reg.Register(
 					httpmock.QueryMatcher("GET", "search/commits", values),
-					httpmock.JSONResponse(CommitsResult{
-						IncompleteResults: false,
-						Items:             []Commit{{Sha: "abc"}},
-						Total:             1,
+					httpmock.JSONResponse(map[string]interface{}{
+						"incomplete_results": false,
+						"total_count":        1,
+						"items": []interface{}{
+							map[string]interface{}{
+								"sha": "abc",
+							},
+						},
 					}),
 				)
 			},
@@ -221,10 +293,14 @@ func TestSearcherCommits(t *testing.T) {
 			httpStubs: func(reg *httpmock.Registry) {
 				reg.Register(
 					httpmock.QueryMatcher("GET", "api/v3/search/commits", values),
-					httpmock.JSONResponse(CommitsResult{
-						IncompleteResults: false,
-						Items:             []Commit{{Sha: "abc"}},
-						Total:             1,
+					httpmock.JSONResponse(map[string]interface{}{
+						"incomplete_results": false,
+						"total_count":        1,
+						"items": []interface{}{
+							map[string]interface{}{
+								"sha": "abc",
+							},
+						},
 					}),
 				)
 			},
@@ -239,27 +315,92 @@ func TestSearcherCommits(t *testing.T) {
 			},
 			httpStubs: func(reg *httpmock.Registry) {
 				firstReq := httpmock.QueryMatcher("GET", "search/commits", values)
-				firstRes := httpmock.JSONResponse(CommitsResult{
-					IncompleteResults: false,
-					Items:             []Commit{{Sha: "abc"}},
-					Total:             2,
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        2,
+					"items": []interface{}{
+						map[string]interface{}{
+							"sha": "abc",
+						},
+					},
+				})
+				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/commits?page=2&per_page=30&q=org%3Agithub>; rel="next"`)
+				secondReq := httpmock.QueryMatcher("GET", "search/commits", url.Values{
+					"page":     []string{"2"},
+					"per_page": []string{"30"},
+					"order":    []string{"desc"},
+					"sort":     []string{"committer-date"},
+					"q":        []string{"keyword author:foobar committer-date:>2021-02-28"},
+				})
+				secondRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        2,
+					"items": []interface{}{
+						map[string]interface{}{
+							"sha": "def",
+						},
+					},
+				})
+				reg.Register(firstReq, firstRes)
+				reg.Register(secondReq, secondRes)
+			},
+		},
+		{
+			name: "collect full and partial pages under total number of matching search results",
+			query: Query{
+				Keywords: []string{"keyword"},
+				Kind:     "commits",
+				Limit:    110,
+				Order:    "desc",
+				Sort:     "committer-date",
+				Qualifiers: Qualifiers{
+					Author:        "foobar",
+					CommitterDate: ">2021-02-28",
 				},
-				)
+			},
+			result: CommitsResult{
+				IncompleteResults: false,
+				Items: initialize(0, 110, func(i int) Commit {
+					return Commit{
+						Sha: strconv.Itoa(i),
+					}
+				}),
+				Total: 287,
+			},
+			httpStubs: func(reg *httpmock.Registry) {
+				firstReq := httpmock.QueryMatcher("GET", "search/commits", url.Values{
+					"page":     []string{"1"},
+					"per_page": []string{"100"},
+					"order":    []string{"desc"},
+					"sort":     []string{"committer-date"},
+					"q":        []string{"keyword author:foobar committer-date:>2021-02-28"},
+				})
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(0, 100, func(i int) map[string]interface{} {
+						return map[string]interface{}{
+							"sha": strconv.Itoa(i),
+						}
+					}),
+				})
 				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/commits?page=2&per_page=100&q=org%3Agithub>; rel="next"`)
 				secondReq := httpmock.QueryMatcher("GET", "search/commits", url.Values{
 					"page":     []string{"2"},
-					"per_page": []string{"29"},
+					"per_page": []string{"100"},
 					"order":    []string{"desc"},
 					"sort":     []string{"committer-date"},
 					"q":        []string{"keyword author:foobar committer-date:>2021-02-28"},
-				},
-				)
-				secondRes := httpmock.JSONResponse(CommitsResult{
-					IncompleteResults: false,
-					Items:             []Commit{{Sha: "def"}},
-					Total:             2,
-				},
-				)
+				})
+				secondRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(100, 200, func(i int) map[string]interface{} {
+						return map[string]interface{}{
+							"sha": strconv.Itoa(i),
+						}
+					}),
+				})
 				reg.Register(firstReq, firstRes)
 				reg.Register(secondReq, secondRes)
 			},
@@ -269,8 +410,8 @@ func TestSearcherCommits(t *testing.T) {
 			query:   query,
 			wantErr: true,
 			errMsg: heredoc.Doc(`
-        Invalid search query "keyword author:foobar committer-date:>2021-02-28".
-        "blah" is not a recognized date/time format. Please provide an ISO 8601 date/time value, such as YYYY-MM-DD.`),
+				Invalid search query "keyword author:foobar committer-date:>2021-02-28".
+				"blah" is not a recognized date/time format. Please provide an ISO 8601 date/time value, such as YYYY-MM-DD.`),
 			httpStubs: func(reg *httpmock.Registry) {
 				reg.Register(
 					httpmock.QueryMatcher("GET", "search/commits", values),
@@ -413,15 +554,14 @@ func TestSearcherRepositories(t *testing.T) {
 						},
 					},
 				})
-				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/repositories?page=2&per_page=100&q=org%3Agithub>; rel="next"`)
+				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/repositories?page=2&per_page=30&q=org%3Agithub>; rel="next"`)
 				secondReq := httpmock.QueryMatcher("GET", "search/repositories", url.Values{
 					"page":     []string{"2"},
-					"per_page": []string{"29"},
+					"per_page": []string{"30"},
 					"order":    []string{"desc"},
 					"sort":     []string{"stars"},
 					"q":        []string{"keyword stars:>=5 topic:topic"},
-				},
-				)
+				})
 				secondRes := httpmock.JSONResponse(map[string]interface{}{
 					"incomplete_results": false,
 					"total_count":        2,
@@ -435,13 +575,73 @@ func TestSearcherRepositories(t *testing.T) {
 				reg.Register(secondReq, secondRes)
 			},
 		},
+		{
+			name: "collect full and partial pages under total number of matching search results",
+			query: Query{
+				Keywords: []string{"keyword"},
+				Kind:     "repositories",
+				Limit:    110,
+				Order:    "desc",
+				Sort:     "stars",
+				Qualifiers: Qualifiers{
+					Stars: ">=5",
+					Topic: []string{"topic"},
+				},
+			},
+			result: RepositoriesResult{
+				IncompleteResults: false,
+				Items: initialize(0, 110, func(i int) Repository {
+					return Repository{
+						Name: fmt.Sprintf("name%d", i),
+					}
+				}),
+				Total: 287,
+			},
+			httpStubs: func(reg *httpmock.Registry) {
+				firstReq := httpmock.QueryMatcher("GET", "search/repositories", url.Values{
+					"page":     []string{"1"},
+					"per_page": []string{"100"},
+					"order":    []string{"desc"},
+					"sort":     []string{"stars"},
+					"q":        []string{"keyword stars:>=5 topic:topic"},
+				})
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(0, 100, func(i int) interface{} {
+						return map[string]interface{}{
+							"name": fmt.Sprintf("name%d", i),
+						}
+					}),
+				})
+				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/repositories?page=2&per_page=100&q=org%3Agithub>; rel="next"`)
+				secondReq := httpmock.QueryMatcher("GET", "search/repositories", url.Values{
+					"page":     []string{"2"},
+					"per_page": []string{"100"},
+					"order":    []string{"desc"},
+					"sort":     []string{"stars"},
+					"q":        []string{"keyword stars:>=5 topic:topic"},
+				})
+				secondRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(100, 200, func(i int) interface{} {
+						return map[string]interface{}{
+							"name": fmt.Sprintf("name%d", i),
+						}
+					}),
+				})
+				reg.Register(firstReq, firstRes)
+				reg.Register(secondReq, secondRes)
+			},
+		},
 		{
 			name:    "handles search errors",
 			query:   query,
 			wantErr: true,
 			errMsg: heredoc.Doc(`
-        Invalid search query "keyword stars:>=5 topic:topic".
-        "blah" is not a recognized date/time format. Please provide an ISO 8601 date/time value, such as YYYY-MM-DD.`),
+				Invalid search query "keyword stars:>=5 topic:topic".
+				"blah" is not a recognized date/time format. Please provide an ISO 8601 date/time value, such as YYYY-MM-DD.`),
 			httpStubs: func(reg *httpmock.Registry) {
 				reg.Register(
 					httpmock.QueryMatcher("GET", "search/repositories", values),
@@ -529,10 +729,14 @@ func TestSearcherIssues(t *testing.T) {
 			httpStubs: func(reg *httpmock.Registry) {
 				reg.Register(
 					httpmock.QueryMatcher("GET", "search/issues", values),
-					httpmock.JSONResponse(IssuesResult{
-						IncompleteResults: false,
-						Items:             []Issue{{Number: 1234}},
-						Total:             1,
+					httpmock.JSONResponse(map[string]interface{}{
+						"incomplete_results": false,
+						"total_count":        1,
+						"items": []interface{}{
+							map[string]interface{}{
+								"number": 1234,
+							},
+						},
 					}),
 				)
 			},
@@ -549,10 +753,14 @@ func TestSearcherIssues(t *testing.T) {
 			httpStubs: func(reg *httpmock.Registry) {
 				reg.Register(
 					httpmock.QueryMatcher("GET", "api/v3/search/issues", values),
-					httpmock.JSONResponse(IssuesResult{
-						IncompleteResults: false,
-						Items:             []Issue{{Number: 1234}},
-						Total:             1,
+					httpmock.JSONResponse(map[string]interface{}{
+						"incomplete_results": false,
+						"total_count":        1,
+						"items": []interface{}{
+							map[string]interface{}{
+								"number": 1234,
+							},
+						},
 					}),
 				)
 			},
@@ -567,27 +775,92 @@ func TestSearcherIssues(t *testing.T) {
 			},
 			httpStubs: func(reg *httpmock.Registry) {
 				firstReq := httpmock.QueryMatcher("GET", "search/issues", values)
-				firstRes := httpmock.JSONResponse(IssuesResult{
-					IncompleteResults: false,
-					Items:             []Issue{{Number: 1234}},
-					Total:             2,
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        2,
+					"items": []interface{}{
+						map[string]interface{}{
+							"number": 1234,
+						},
+					},
+				})
+				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/issues?page=2&per_page=30&q=org%3Agithub>; rel="next"`)
+				secondReq := httpmock.QueryMatcher("GET", "search/issues", url.Values{
+					"page":     []string{"2"},
+					"per_page": []string{"30"},
+					"order":    []string{"desc"},
+					"sort":     []string{"comments"},
+					"q":        []string{"keyword is:locked is:public language:go"},
+				})
+				secondRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        2,
+					"items": []interface{}{
+						map[string]interface{}{
+							"number": 5678,
+						},
+					},
+				})
+				reg.Register(firstReq, firstRes)
+				reg.Register(secondReq, secondRes)
+			},
+		},
+		{
+			name: "collect full and partial pages under total number of matching search results",
+			query: Query{
+				Keywords: []string{"keyword"},
+				Kind:     "issues",
+				Limit:    110,
+				Order:    "desc",
+				Sort:     "comments",
+				Qualifiers: Qualifiers{
+					Language: "go",
+					Is:       []string{"public", "locked"},
 				},
-				)
+			},
+			result: IssuesResult{
+				IncompleteResults: false,
+				Items: initialize(0, 110, func(i int) Issue {
+					return Issue{
+						Number: i,
+					}
+				}),
+				Total: 287,
+			},
+			httpStubs: func(reg *httpmock.Registry) {
+				firstReq := httpmock.QueryMatcher("GET", "search/issues", url.Values{
+					"page":     []string{"1"},
+					"per_page": []string{"100"},
+					"order":    []string{"desc"},
+					"sort":     []string{"comments"},
+					"q":        []string{"keyword is:locked is:public language:go"},
+				})
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(0, 100, func(i int) interface{} {
+						return map[string]interface{}{
+							"number": i,
+						}
+					}),
+				})
 				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/issues?page=2&per_page=100&q=org%3Agithub>; rel="next"`)
 				secondReq := httpmock.QueryMatcher("GET", "search/issues", url.Values{
 					"page":     []string{"2"},
-					"per_page": []string{"29"},
+					"per_page": []string{"100"},
 					"order":    []string{"desc"},
 					"sort":     []string{"comments"},
 					"q":        []string{"keyword is:locked is:public language:go"},
-				},
-				)
-				secondRes := httpmock.JSONResponse(IssuesResult{
-					IncompleteResults: false,
-					Items:             []Issue{{Number: 5678}},
-					Total:             2,
-				},
-				)
+				})
+				secondRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(100, 200, func(i int) interface{} {
+						return map[string]interface{}{
+							"number": i,
+						}
+					}),
+				})
 				reg.Register(firstReq, firstRes)
 				reg.Register(secondReq, secondRes)
 			},
@@ -597,8 +870,8 @@ func TestSearcherIssues(t *testing.T) {
 			query:   query,
 			wantErr: true,
 			errMsg: heredoc.Doc(`
-        Invalid search query "keyword is:locked is:public language:go".
-        "blah" is not a recognized date/time format. Please provide an ISO 8601 date/time value, such as YYYY-MM-DD.`),
+				Invalid search query "keyword is:locked is:public language:go".
+				"blah" is not a recognized date/time format. Please provide an ISO 8601 date/time value, such as YYYY-MM-DD.`),
 			httpStubs: func(reg *httpmock.Registry) {
 				reg.Register(
 					httpmock.QueryMatcher("GET", "search/issues", values),
@@ -686,3 +959,12 @@ func TestSearcherURL(t *testing.T) {
 		})
 	}
 }
+
+// initialize generate slices over a range for test scenarios using the provided initializer.
+func initialize[T any](start int, stop int, initializer func(i int) T) []T {
+	results := make([]T, 0, (stop - start))
+	for i := start; i < stop; i++ {
+		results = append(results, initializer(i))
+	}
+	return results
+}


Reviews:
user: login: leudz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T02:05:22Z
user: login: andyfeller
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T20:42:20Z
user: login: andyfeller
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T12:05:03Z
user: login: andyfeller
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T18:50:28Z
user: login: andyfeller
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T21:34:16Z
user: login: leudz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T23:42:25Z
user: login: leudz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T23:43:52Z
user: login: andyfeller
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T11:45:50Z
user: login: andyfeller
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T17:34:14Z
user: login: BagToad
body: LGTM! ✨ The problem and fix make sense, and this branch does fix the problem when I tested it out myself.  
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-16T20:54:16Z

Review comments:
diff_hunk: @@ -60,96 +61,138 @@ func NewSearcher(client *http.Client, host string) Searcher {
 
 func (s searcher) Code(query Query) (CodeResult, error) {
 	result := CodeResult{}
-	toRetrieve := query.Limit
+
 	var resp *http.Response
 	var err error
+
+	toRetrieve := query.Limit
+	// We will request either the query limit if it's less than 1 page, or our max page size.
+	// This number doesn't change to keep a valid offset.
+	//
+	// For example, say we want 150 items out of 500.
+	// We request page #1 for 100 items and get items 0 to 99.
+	// Then we request page #2 for 100 items, we get items 100 to 199 and only keep 100 to 149.
+	// If we were to request page #2 for 50 items, we would instead get items 50 to 99.
+	query.Limit = min(toRetrieve, maxPerPage)
+
 	for toRetrieve > 0 {
-		query.Limit = min(toRetrieve, maxPerPage)
 		query.Page = nextPage(resp)
 		if query.Page == 0 {
 			break
 		}
+
 		page := CodeResult{}
 		resp, err = s.search(query, &page)
 		if err != nil {
 			return result, err
 		}
+
 		result.IncompleteResults = page.IncompleteResults
-		result.Total = page.Total
-		result.Items = append(result.Items, page.Items...)
-		toRetrieve = toRetrieve - len(page.Items)
+
+		// If we're going to reach the requested limit, only add that many items,
+		// otherwise add all the results.
+		itemsToAdd := min(len(page.Items), toRetrieve)
+
+		result.Total += itemsToAdd
path: pkg/search/searcher.go
user: login: leudz
body: I'm not 100% sure what this total is.
I assume it's the total number of elements returned.

It used to only return the last page's total which affect the pagination test.
created_at: 2025-04-11T01:56:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -87,20 +103,64 @@ func TestSearcherCode(t *testing.T) {
 				firstRes := httpmock.JSONResponse(CodeResult{
 					IncompleteResults: false,
 					Items:             []Code{{Name: "file.go"}},
-					Total:             2,
+					Total:             1,
 				},
 				)
-				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/code?page=2&per_page=100&q=org%3Agithub>; rel="next"`)
+				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/code?page=2&per_page=30&q=org%3Agithub>; rel="next"`)
 				secondReq := httpmock.QueryMatcher("GET", "search/code", url.Values{
 					"page":     []string{"2"},
-					"per_page": []string{"29"},
+					"per_page": []string{"30"},
 					"q":        []string{"keyword language:go"},
 				},
 				)
 				secondRes := httpmock.JSONResponse(CodeResult{
 					IncompleteResults: false,
 					Items:             []Code{{Name: "file2.go"}},
-					Total:             2,
+					Total:             1,
+				},
+				)
+				reg.Register(firstReq, firstRes)
+				reg.Register(secondReq, secondRes)
+			},
+		},
path: pkg/search/searcher_test.go
user: login: leudz
body: There are only 4 small changes in this tests.
`per_page` is 30, both in the header and the second request.

The total for the first and second responses have gone from 2 to 1.
This link to a previous review comment. I assume this total is the number of items in the response.
In that case, it has always been 1.
created_at: 2025-04-11T02:02:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -60,96 +61,138 @@ func NewSearcher(client *http.Client, host string) Searcher {
 
 func (s searcher) Code(query Query) (CodeResult, error) {
 	result := CodeResult{}
-	toRetrieve := query.Limit
+
 	var resp *http.Response
 	var err error
+
+	toRetrieve := query.Limit
+	// We will request either the query limit if it's less than 1 page, or our max page size.
+	// This number doesn't change to keep a valid offset.
+	//
+	// For example, say we want 150 items out of 500.
+	// We request page #1 for 100 items and get items 0 to 99.
+	// Then we request page #2 for 100 items, we get items 100 to 199 and only keep 100 to 149.
+	// If we were to request page #2 for 50 items, we would instead get items 50 to 99.
+	query.Limit = min(toRetrieve, maxPerPage)
+
 	for toRetrieve > 0 {
-		query.Limit = min(toRetrieve, maxPerPage)
 		query.Page = nextPage(resp)
 		if query.Page == 0 {
 			break
 		}
+
 		page := CodeResult{}
 		resp, err = s.search(query, &page)
 		if err != nil {
 			return result, err
 		}
+
 		result.IncompleteResults = page.IncompleteResults
-		result.Total = page.Total
-		result.Items = append(result.Items, page.Items...)
-		toRetrieve = toRetrieve - len(page.Items)
+
+		// If we're going to reach the requested limit, only add that many items,
+		// otherwise add all the results.
+		itemsToAdd := min(len(page.Items), toRetrieve)
+
+		result.Total += itemsToAdd
path: pkg/search/searcher.go
user: login: andyfeller
body: `Total` is used in displaying the results for communicating how many results out of the total results are being shown.

For example, `gh search code`:

https://github.com/cli/cli/blob/072534c388cf20924be3e359819c2674d6d4190a/pkg/cmd/search/code/code.go#L136-L163
created_at: 2025-04-14T20:42:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -60,96 +61,138 @@ func NewSearcher(client *http.Client, host string) Searcher {
 
 func (s searcher) Code(query Query) (CodeResult, error) {
 	result := CodeResult{}
-	toRetrieve := query.Limit
+
 	var resp *http.Response
 	var err error
+
+	toRetrieve := query.Limit
+	// We will request either the query limit if it's less than 1 page, or our max page size.
+	// This number doesn't change to keep a valid offset.
+	//
+	// For example, say we want 150 items out of 500.
+	// We request page #1 for 100 items and get items 0 to 99.
+	// Then we request page #2 for 100 items, we get items 100 to 199 and only keep 100 to 149.
+	// If we were to request page #2 for 50 items, we would instead get items 50 to 99.
+	query.Limit = min(toRetrieve, maxPerPage)
+
 	for toRetrieve > 0 {
-		query.Limit = min(toRetrieve, maxPerPage)
 		query.Page = nextPage(resp)
 		if query.Page == 0 {
 			break
 		}
+
 		page := CodeResult{}
 		resp, err = s.search(query, &page)
 		if err != nil {
 			return result, err
 		}
+
 		result.IncompleteResults = page.IncompleteResults
-		result.Total = page.Total
-		result.Items = append(result.Items, page.Items...)
-		toRetrieve = toRetrieve - len(page.Items)
+
+		// If we're going to reach the requested limit, only add that many items,
+		// otherwise add all the results.
+		itemsToAdd := min(len(page.Items), toRetrieve)
+
+		result.Total += itemsToAdd
path: pkg/search/searcher.go
user: login: andyfeller
body: **issue:** I have reverted this logic because it causes the `X of Y` results header to report `--limit` for Y instead of the total number of matching search results.

```
		result.Total = page.Total
```

### Demo

Below, we see this 150-limited code search reporting exactly 150 total matches are found:

```shell
$ bin/gh search code "fmt.Fprintf" --repo cli/cli --limit 150

Showing 150 of 150 results

cli/cli script/build.go
	fmt.Fprintf(os.Stderr, "Don't know how to build task `%s`.\n", task)
	fmt.Fprintf(os.Stderr, "%s: building task `%s` failed.\n", self, task)
```

with original logic, we see `150 of 161 results` which is the total number of matching search entries:

```shell
$ bin/gh search code "fmt.Fprintf" --repo cli/cli --limit 150

Showing 150 of 161 results

cli/cli script/build.go
	fmt.Fprintf(os.Stderr, "Don't know how to build task `%s`.\n", task)
	fmt.Fprintf(os.Stderr, "%s: building task `%s` failed.\n", self, task)
```
created_at: 2025-04-15T12:05:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -87,20 +103,64 @@ func TestSearcherCode(t *testing.T) {
 				firstRes := httpmock.JSONResponse(CodeResult{
 					IncompleteResults: false,
 					Items:             []Code{{Name: "file.go"}},
-					Total:             2,
+					Total:             1,
 				},
 				)
-				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/code?page=2&per_page=100&q=org%3Agithub>; rel="next"`)
+				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/code?page=2&per_page=30&q=org%3Agithub>; rel="next"`)
 				secondReq := httpmock.QueryMatcher("GET", "search/code", url.Values{
 					"page":     []string{"2"},
-					"per_page": []string{"29"},
+					"per_page": []string{"30"},
 					"q":        []string{"keyword language:go"},
 				},
 				)
 				secondRes := httpmock.JSONResponse(CodeResult{
 					IncompleteResults: false,
 					Items:             []Code{{Name: "file2.go"}},
-					Total:             2,
+					Total:             1,
+				},
+				)
+				reg.Register(firstReq, firstRes)
+				reg.Register(secondReq, secondRes)
+			},
+		},
path: pkg/search/searcher_test.go
user: login: andyfeller
body: @leudz : As the `.Total` logic changes needed to be reverted, there are going to be some changes to these tests as `.Total` is not the number of items in the page, but the total number of matches to the search query.

Additionally, I've made some enhancements around test table data initialization.
created_at: 2025-04-15T18:50:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -435,13 +575,73 @@ func TestSearcherRepositories(t *testing.T) {
 				reg.Register(secondReq, secondRes)
 			},
 		},
+		{
+			name: "collect full and partial pages under total number of matching search results",
+			query: Query{
+				Keywords: []string{"keyword"},
+				Kind:     "repositories",
+				Limit:    110,
+				Order:    "desc",
+				Sort:     "stars",
+				Qualifiers: Qualifiers{
+					Stars: ">=5",
+					Topic: []string{"topic"},
+				},
+			},
+			result: RepositoriesResult{
+				IncompleteResults: false,
+				Items: initialize(0, 110, func(i int) Repository {
+					return Repository{
+						Name: fmt.Sprintf("name%d", i),
+					}
+				}),
+				Total: 287,
+			},
+			httpStubs: func(reg *httpmock.Registry) {
+				firstReq := httpmock.QueryMatcher("GET", "search/repositories", url.Values{
+					"page":     []string{"1"},
+					"per_page": []string{"100"},
+					"order":    []string{"desc"},
+					"sort":     []string{"stars"},
+					"q":        []string{"keyword stars:>=5 topic:topic"},
+				})
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(0, 100, func(i int) interface{} {
+						return map[string]interface{}{
+							"name": fmt.Sprintf("name%d", i),
+						}
+					}),
+				})
+				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/repositories?page=2&per_page=100&q=org%3Agithub>; rel="next"`)
+				secondReq := httpmock.QueryMatcher("GET", "search/repositories", url.Values{
+					"page":     []string{"2"},
+					"per_page": []string{"100"},
+					"order":    []string{"desc"},
+					"sort":     []string{"stars"},
+					"q":        []string{"keyword stars:>=5 topic:topic"},
+				})
+				secondRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(100, 200, func(i int) interface{} {
+						return map[string]interface{}{
+							"name": fmt.Sprintf("name%d", i),
+						}
+					}),
+				})
+				reg.Register(firstReq, firstRes)
+				reg.Register(secondReq, secondRes)
+			},
+		},
path: pkg/search/searcher_test.go
user: login: andyfeller
body: **concern:** originally, these tests relied upon the `httpmock.Registry` to JSON marshal the `*Result` types to JSON, however there is something about the `search.Repository` type where `Name` field was not being preserved in the marshaling.

Rather than having inconsistent test setups, I aligned the tests on capturing the JSON we expect from the REST API.
created_at: 2025-04-15T21:29:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -435,13 +575,73 @@ func TestSearcherRepositories(t *testing.T) {
 				reg.Register(secondReq, secondRes)
 			},
 		},
+		{
+			name: "collect full and partial pages under total number of matching search results",
+			query: Query{
+				Keywords: []string{"keyword"},
+				Kind:     "repositories",
+				Limit:    110,
+				Order:    "desc",
+				Sort:     "stars",
+				Qualifiers: Qualifiers{
+					Stars: ">=5",
+					Topic: []string{"topic"},
+				},
+			},
+			result: RepositoriesResult{
+				IncompleteResults: false,
+				Items: initialize(0, 110, func(i int) Repository {
+					return Repository{
+						Name: fmt.Sprintf("name%d", i),
+					}
+				}),
+				Total: 287,
+			},
+			httpStubs: func(reg *httpmock.Registry) {
+				firstReq := httpmock.QueryMatcher("GET", "search/repositories", url.Values{
+					"page":     []string{"1"},
+					"per_page": []string{"100"},
+					"order":    []string{"desc"},
+					"sort":     []string{"stars"},
+					"q":        []string{"keyword stars:>=5 topic:topic"},
+				})
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(0, 100, func(i int) interface{} {
+						return map[string]interface{}{
+							"name": fmt.Sprintf("name%d", i),
+						}
+					}),
+				})
+				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/repositories?page=2&per_page=100&q=org%3Agithub>; rel="next"`)
+				secondReq := httpmock.QueryMatcher("GET", "search/repositories", url.Values{
+					"page":     []string{"2"},
+					"per_page": []string{"100"},
+					"order":    []string{"desc"},
+					"sort":     []string{"stars"},
+					"q":        []string{"keyword stars:>=5 topic:topic"},
+				})
+				secondRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(100, 200, func(i int) interface{} {
path: pkg/search/searcher_test.go
user: login: andyfeller
body: **thought:** while discussing these tests with @jtmcg, Tyler made a call out how the REST API will return a full second page.  He suggested initializing this to be a full page, which is closer to the behavior we will encounter in use.
created_at: 2025-04-15T21:31:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -435,13 +575,73 @@ func TestSearcherRepositories(t *testing.T) {
 				reg.Register(secondReq, secondRes)
 			},
 		},
+		{
+			name: "collect full and partial pages under total number of matching search results",
path: pkg/search/searcher_test.go
user: login: andyfeller
body: **thought:** after talking with @jtmcg, Tyler suggested renaming these test scenarios to explain they are ensuring these commands process REST responses and return a collection of full and partial pages to the user while still being beneath the total number of matching search results.
created_at: 2025-04-15T21:32:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -239,27 +315,92 @@ func TestSearcherCommits(t *testing.T) {
 			},
 			httpStubs: func(reg *httpmock.Registry) {
 				firstReq := httpmock.QueryMatcher("GET", "search/commits", values)
-				firstRes := httpmock.JSONResponse(CommitsResult{
-					IncompleteResults: false,
-					Items:             []Commit{{Sha: "abc"}},
-					Total:             2,
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        2,
+					"items": []interface{}{
+						map[string]interface{}{
+							"sha": "abc",
+						},
+					},
+				})
+				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/commits?page=2&per_page=30&q=org%3Agithub>; rel="next"`)
+				secondReq := httpmock.QueryMatcher("GET", "search/commits", url.Values{
+					"page":     []string{"2"},
+					"per_page": []string{"30"},
+					"order":    []string{"desc"},
+					"sort":     []string{"committer-date"},
+					"q":        []string{"keyword author:foobar committer-date:>2021-02-28"},
+				})
+				secondRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        2,
+					"items": []interface{}{
+						map[string]interface{}{
+							"sha": "def",
+						},
+					},
+				})
+				reg.Register(firstReq, firstRes)
+				reg.Register(secondReq, secondRes)
+			},
+		},
+		{
+			name: "collect full and partial pages under total number of matching search results",
+			query: Query{
+				Keywords: []string{"keyword"},
+				Kind:     "commits",
+				Limit:    110,
+				Order:    "desc",
+				Sort:     "committer-date",
+				Qualifiers: Qualifiers{
+					Author:        "foobar",
+					CommitterDate: ">2021-02-28",
 				},
-				)
+			},
+			result: CommitsResult{
+				IncompleteResults: false,
+				Items: initialize(0, 110, func(i int) Commit {
+					return Commit{
+						Sha: strconv.Itoa(i),
+					}
+				}),
+				Total: 287,
+			},
+			httpStubs: func(reg *httpmock.Registry) {
+				firstReq := httpmock.QueryMatcher("GET", "search/commits", url.Values{
+					"page":     []string{"1"},
+					"per_page": []string{"100"},
+					"order":    []string{"desc"},
+					"sort":     []string{"committer-date"},
+					"q":        []string{"keyword author:foobar committer-date:>2021-02-28"},
+				})
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
path: pkg/search/searcher_test.go
user: login: andyfeller
body: **thought:** the REST search API returns the total number of matching search results on every page.  In order to have a number stand out separate from the client-side limit, Tyler suggested 287, which is as good as any.
created_at: 2025-04-15T21:33:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -60,96 +61,138 @@ func NewSearcher(client *http.Client, host string) Searcher {
 
 func (s searcher) Code(query Query) (CodeResult, error) {
 	result := CodeResult{}
-	toRetrieve := query.Limit
+
 	var resp *http.Response
 	var err error
+
+	toRetrieve := query.Limit
+	// We will request either the query limit if it's less than 1 page, or our max page size.
+	// This number doesn't change to keep a valid offset.
+	//
+	// For example, say we want 150 items out of 500.
+	// We request page #1 for 100 items and get items 0 to 99.
+	// Then we request page #2 for 100 items, we get items 100 to 199 and only keep 100 to 149.
+	// If we were to request page #2 for 50 items, we would instead get items 50 to 99.
+	query.Limit = min(toRetrieve, maxPerPage)
+
 	for toRetrieve > 0 {
-		query.Limit = min(toRetrieve, maxPerPage)
 		query.Page = nextPage(resp)
 		if query.Page == 0 {
 			break
 		}
+
 		page := CodeResult{}
 		resp, err = s.search(query, &page)
 		if err != nil {
 			return result, err
 		}
+
 		result.IncompleteResults = page.IncompleteResults
-		result.Total = page.Total
-		result.Items = append(result.Items, page.Items...)
-		toRetrieve = toRetrieve - len(page.Items)
+
+		// If we're going to reach the requested limit, only add that many items,
+		// otherwise add all the results.
+		itemsToAdd := min(len(page.Items), toRetrieve)
+
+		result.Total += itemsToAdd
path: pkg/search/searcher.go
user: login: leudz
body: Okay so it's the total matching the search query... sorry for the useless changes. Would it be worth add a small comment on the field?
created_at: 2025-04-15T23:42:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -239,27 +315,92 @@ func TestSearcherCommits(t *testing.T) {
 			},
 			httpStubs: func(reg *httpmock.Registry) {
 				firstReq := httpmock.QueryMatcher("GET", "search/commits", values)
-				firstRes := httpmock.JSONResponse(CommitsResult{
-					IncompleteResults: false,
-					Items:             []Commit{{Sha: "abc"}},
-					Total:             2,
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        2,
+					"items": []interface{}{
+						map[string]interface{}{
+							"sha": "abc",
+						},
+					},
+				})
+				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/commits?page=2&per_page=30&q=org%3Agithub>; rel="next"`)
+				secondReq := httpmock.QueryMatcher("GET", "search/commits", url.Values{
+					"page":     []string{"2"},
+					"per_page": []string{"30"},
+					"order":    []string{"desc"},
+					"sort":     []string{"committer-date"},
+					"q":        []string{"keyword author:foobar committer-date:>2021-02-28"},
+				})
+				secondRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        2,
+					"items": []interface{}{
+						map[string]interface{}{
+							"sha": "def",
+						},
+					},
+				})
+				reg.Register(firstReq, firstRes)
+				reg.Register(secondReq, secondRes)
+			},
+		},
+		{
+			name: "collect full and partial pages under total number of matching search results",
+			query: Query{
+				Keywords: []string{"keyword"},
+				Kind:     "commits",
+				Limit:    110,
+				Order:    "desc",
+				Sort:     "committer-date",
+				Qualifiers: Qualifiers{
+					Author:        "foobar",
+					CommitterDate: ">2021-02-28",
 				},
-				)
+			},
+			result: CommitsResult{
+				IncompleteResults: false,
+				Items: initialize(0, 110, func(i int) Commit {
+					return Commit{
+						Sha: strconv.Itoa(i),
+					}
+				}),
+				Total: 287,
+			},
+			httpStubs: func(reg *httpmock.Registry) {
+				firstReq := httpmock.QueryMatcher("GET", "search/commits", url.Values{
+					"page":     []string{"1"},
+					"per_page": []string{"100"},
+					"order":    []string{"desc"},
+					"sort":     []string{"committer-date"},
+					"q":        []string{"keyword author:foobar committer-date:>2021-02-28"},
+				})
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
path: pkg/search/searcher_test.go
user: login: leudz
body: That's a very nice change 👍 
created_at: 2025-04-15T23:43:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -60,96 +61,138 @@ func NewSearcher(client *http.Client, host string) Searcher {
 
 func (s searcher) Code(query Query) (CodeResult, error) {
 	result := CodeResult{}
-	toRetrieve := query.Limit
+
 	var resp *http.Response
 	var err error
+
+	toRetrieve := query.Limit
+	// We will request either the query limit if it's less than 1 page, or our max page size.
+	// This number doesn't change to keep a valid offset.
+	//
+	// For example, say we want 150 items out of 500.
+	// We request page #1 for 100 items and get items 0 to 99.
+	// Then we request page #2 for 100 items, we get items 100 to 199 and only keep 100 to 149.
+	// If we were to request page #2 for 50 items, we would instead get items 50 to 99.
+	query.Limit = min(toRetrieve, maxPerPage)
+
 	for toRetrieve > 0 {
-		query.Limit = min(toRetrieve, maxPerPage)
 		query.Page = nextPage(resp)
 		if query.Page == 0 {
 			break
 		}
+
 		page := CodeResult{}
 		resp, err = s.search(query, &page)
 		if err != nil {
 			return result, err
 		}
+
 		result.IncompleteResults = page.IncompleteResults
-		result.Total = page.Total
-		result.Items = append(result.Items, page.Items...)
-		toRetrieve = toRetrieve - len(page.Items)
+
+		// If we're going to reach the requested limit, only add that many items,
+		// otherwise add all the results.
+		itemsToAdd := min(len(page.Items), toRetrieve)
+
+		result.Total += itemsToAdd
path: pkg/search/searcher.go
user: login: andyfeller
body: I think that is a solid call out as it really isn't obvious, yeah 💯 
created_at: 2025-04-16T11:45:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -435,13 +575,73 @@ func TestSearcherRepositories(t *testing.T) {
 				reg.Register(secondReq, secondRes)
 			},
 		},
+		{
+			name: "collect full and partial pages under total number of matching search results",
+			query: Query{
+				Keywords: []string{"keyword"},
+				Kind:     "repositories",
+				Limit:    110,
+				Order:    "desc",
+				Sort:     "stars",
+				Qualifiers: Qualifiers{
+					Stars: ">=5",
+					Topic: []string{"topic"},
+				},
+			},
+			result: RepositoriesResult{
+				IncompleteResults: false,
+				Items: initialize(0, 110, func(i int) Repository {
+					return Repository{
+						Name: fmt.Sprintf("name%d", i),
+					}
+				}),
+				Total: 287,
+			},
+			httpStubs: func(reg *httpmock.Registry) {
+				firstReq := httpmock.QueryMatcher("GET", "search/repositories", url.Values{
+					"page":     []string{"1"},
+					"per_page": []string{"100"},
+					"order":    []string{"desc"},
+					"sort":     []string{"stars"},
+					"q":        []string{"keyword stars:>=5 topic:topic"},
+				})
+				firstRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(0, 100, func(i int) interface{} {
+						return map[string]interface{}{
+							"name": fmt.Sprintf("name%d", i),
+						}
+					}),
+				})
+				firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/repositories?page=2&per_page=100&q=org%3Agithub>; rel="next"`)
+				secondReq := httpmock.QueryMatcher("GET", "search/repositories", url.Values{
+					"page":     []string{"2"},
+					"per_page": []string{"100"},
+					"order":    []string{"desc"},
+					"sort":     []string{"stars"},
+					"q":        []string{"keyword stars:>=5 topic:topic"},
+				})
+				secondRes := httpmock.JSONResponse(map[string]interface{}{
+					"incomplete_results": false,
+					"total_count":        287,
+					"items": initialize(100, 200, func(i int) interface{} {
+						return map[string]interface{}{
+							"name": fmt.Sprintf("name%d", i),
+						}
+					}),
+				})
+				reg.Register(firstReq, firstRes)
+				reg.Register(secondReq, secondRes)
+			},
+		},
path: pkg/search/searcher_test.go
user: login: andyfeller
body: Closing the loop here, it appears that `search.Repository` JSON marshaling is a problem because of this logic used for exporting `gh search code` as it also contains a repository field:

https://github.com/cli/cli/blob/5d8dbc0d534d71d1b011e52016fe17b6565a5317/pkg/search/result.go#L356-L385

Given this surprising behavior, the move away from using the `Results` type for response mocking makes more sense.
created_at: 2025-04-16T17:34:13Z
author_association: CONTRIBUTOR

PR comments:
user: login: andyfeller
created_at: 2025-04-14T20:43:39Z
body: Thank you for your patience and efforts to improve the GitHub CLI, @leudz! ❤️  I'm working to provide some initial feedback today so we can get this shaped up and shipped out. 💪 
author_association: CONTRIBUTOR
user: login: andyfeller
created_at: 2025-04-15T12:37:12Z
body: ## Demo

After restoring the `.Total` logic, I wanted to manually confirm the experience relative to the GitHub REST Search API results.

### `gh search code`

```shell
$ gh api "search/code?q=fmt.Fprintf+repo:cli/cli" -X GET --jq '.total_count'
161

$ gh search code fmt.Fprintf --repo cli/cli

Showing 30 of 161 results
```

### `gh search commits`

```shell
$ gh api "search/commits?q=author:andyfeller+repo:cli/cli" -X GET --jq '.total_count'
335

$ gh search commits --repo cli/cli author:andyfeller      

Showing 30 of 335 commits
```

### `gh search issues`

```shell
$ gh api "search/issues?q=author:andyfeller+repo:cli/cli+type:issue" -X GET --jq '.total_count'
39

$ gh search issues --repo cli/cli author:andyfeller                                            

Showing 30 of 39 issues
```

### `gh search prs`

```shell
$ gh api "search/issues?q=author:andyfeller+repo:cli/cli+type:pr" -X GET --jq '.total_count'
51

$ gh search prs --repo cli/cli author:andyfeller                                       

Showing 30 of 51 pull requests
```

### `gh search repos`

```shell
$ gh api "search/repositories?q=user:andyfeller" -X GET --jq '.total_count'
65

$ gh search repos --owner andyfeller

Showing 30 of 65 repositories
```
author_association: CONTRIBUTOR
user: login: andyfeller
created_at: 2025-04-16T12:54:00Z
body: @leudz : Just because I did a bit more changes to this PR rather than opening up a ton of suggestions, I've asked a fellow maintainer to be the final PR reviewer as I'd value a 2nd set of eyes. 🙇 Thank you again for your patience!
author_association: CONTRIBUTOR
user: login: leudz
created_at: 2025-04-16T18:49:32Z
body: @andyfeller I've added a few comments for Total like we talked about.
With the doc on `search` you've added it's a bit less valuable but it might still avoid someone from missing it.
author_association: CONTRIBUTOR