open-index/open-npm huggingface dataset:
npm package: @mktcodelib/github-scanner
Description: Fetch and store larger amounts of GitHub data in the browser.
Author: mktcode
Latest version: 0.4.0

# GitHub Scanner

Using [@mktcodelib/graphql-fetch-all](https://npmjs.com/package/@mktcodelib/graphql-fetch-all) and adds [IndexedDB](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API) storage and [Web Workers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) on top of it. Install and use it in your project like this:

```bash
npm i @mktcodelib/github-scanner

cp node_modules/@mktcodelib/github-scanner/dist/github-scan-worker.js public/github-scan-worker.js
```

```js
import { Scanner } from '@mktcodelib/github-scanner';

const scanner = new Scanner({ accessToken: "<access token>" });

// Fetch all data from paginated query
const result = await scanner.scan(query, variables);

// Fetch data with intermediate results using a callback
const result = await scanner.scan(
  query,
  variables,
  (intermediateResult) => {
    const { scanId, data, paginators, variables } = intermediateResult;
    // ...
  }
);

// Continue an unfinished scan
const result = await scanner.continueScan(scanId, (intermediateResult) => {
  // ...
});
```

## Database

GitHub Scanner uses IndexedDB to store the fetched data in the browser. The database schema is defined in the `db.ts` file. [Dexie.js](https://dexie.org/) is being used as a simple and efficient way to interact with IndexedDB.

You can access the database directly using the exported `db` instance.

```js
import { db } from '@mktcodelib/github-scanner';

db.scans.toArray().then((scans) => {
  console.log(scans);
});
```

## Web Workers

Each scan is performed in a web worker to avoid blocking the main thread. The worker file is provided by the package and needs to be copied to a location that is accessible by the browser, e.g.:

```bash
cp node_modules/@mktcodelib/github-scanner/dist/github-scan-worker.js public/github-scan-worker.js
```

If you want to use a different filename or path, you can pass it to the `Scanner` constructor:

```js
const scanner = new Scanner({
  accessToken: "<access token>",
  workerPath: '/path/to/worker.js', // default: '/github-scan-worker.js'
});
```

The web worker file is responsible for fetching the data and storing it in the database. It communicates with the main thread using `postMessage` and `onmessage` to send and receive updates.