helloadhavan/github_issues huggingface dataset:
Repository: ggml-org/llama.cpp
Pull request: tokenize : add --show-count (token) option

This commit adds a new option to the tokenize example, --show-count. When this is set the total number of tokens are printed to stdout.

This was added as an option as I was concerned that there might be scripts that use the output from this program and it might be better to not print this information by default.

The motivation for this is that can be useful to find out how many tokens a file contains, for example when trying to determine prompt input file sizes for testing.



- [x] I have read the [contributing guidelines](https://github.com/ggerganov/llama.cpp/blob/master/CONTRIBUTING.md)
- Self-reported review complexity:
  - [x] Low
  - [ ] Medium
  - [ ] High


Changed files:
filename: examples/tokenize/tokenize.cpp
patch: @@ -30,6 +30,7 @@ static void print_usage_information(const char * argv0, FILE * stream) {
     fprintf(stream, "    --stdin                              read prompt from standard input.\n");
     fprintf(stream, "    --no-bos                             do not ever add a BOS token to the prompt, even if normally the model uses a BOS token.\n");
     fprintf(stream, "    --log-disable                        disable logs. Makes stderr quiet when loading the model.\n");
+    fprintf(stream, "    --show-count                         print the total number of tokens.\n");
 }
 
 static void llama_log_callback_null(ggml_log_level level, const char * text, void * user_data) {
@@ -195,6 +196,7 @@ int main(int raw_argc, char ** raw_argv) {
     bool printing_ids = false;
     bool no_bos = false;
     bool disable_logging = false;
+    bool show_token_count = false;
     const char * model_path = NULL;
     const char * prompt_path = NULL;
     const char * prompt_arg = NULL;
@@ -249,6 +251,9 @@ int main(int raw_argc, char ** raw_argv) {
         else if (arg == "--log-disable") {
             disable_logging = true;
         }
+        else if (arg == "--show-count") {
+            show_token_count = true;
+        }
         else {
             fprintf(stderr, "Error: unknown option '%s'\n", argv[iarg].c_str());
             return 1;
@@ -384,6 +389,9 @@ int main(int raw_argc, char ** raw_argv) {
         printf("]\n");
     }
 
+    if (show_token_count) {
+        printf("Total number of tokens: %ld\n", tokens.size());
+    }
     // silence valgrind
     llama_free(ctx);
     llama_free_model(model);
Repository: facebook/react
Pull request: Handle line endings correctly on Windows in build script for RN

## Summary

We added some post-processing in the build for RN in #26616 that broke for users on Windows due to how line endings were handled to the regular expression to insert some directives in the docblock. This fixes that problem, reported in #26697 as well.

## How did you test this change?

Verified files are still built correctly on Mac/Linux. Will ask for help to test on Windows.

Changed files:
filename: scripts/rollup/packaging.js
patch: @@ -148,9 +148,9 @@ function processGenerated(directory) {
     const originalContents = readFileSync(file, 'utf8');
     const contents = originalContents
       // Replace {@}format with {@}noformat
-      .replace(/(\n\s*\*\s*)@format\b.*(\n)/, '$1@noformat$2')
+      .replace(/(\r?\n\s*\*\s*)@format\b.*(\n)/, '$1@noformat$2')
       // Add {@}nolint and {@}generated
-      .replace(' */\n', ` * @nolint\n * ${getSigningToken()}\n */\n`);
+      .replace(/(\r?\n\s*\*)\//, `$1 @nolint$1 ${getSigningToken()}$1/`);
     const signedContents = signFile(contents);
     writeFileSync(file, signedContents, 'utf8');
   });
Repository: vercel/next.js
Pull request: fix: rspack code analyzer should only handle javascript module

<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:

## For Contributors

### Improving Documentation

- Run `pnpm prettier-fix` to fix formatting issues before opening the PR.
- Read the Docs Contribution Guide to ensure your contribution follows the docs guidelines: https://nextjs.org/docs/community/contribution-guide

### Adding or Updating Examples

- The "examples guidelines" are followed from our contributing doc https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md

### Fixing a bug

- Related issues linked using `fixes #number`
- Tests added. See: https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see https://github.com/vercel/next.js/blob/canary/contributing.md

### Adding a feature

- Implements an existing feature request or RFC. Make sure the feature request has been accepted for implementation before opening a PR. (A discussion must be opened, see https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added (https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see https://github.com/vercel/next.js/blob/canary/contributing.md


## For Maintainers

- Minimal description (aim for explaining to someone not on the team to understand the PR)
- When linking to a Slack thread, you might want to share details of the conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic behind a change

### What?

### Why?

### How?

Closes NEXT-
Fixes #

-->

When import json module in middleware will case error, because codeAnalyzerBySwc should only handle javascript module.


Changed files:
filename: packages/next/src/build/webpack/plugins/middleware-plugin.ts
patch: @@ -583,7 +583,12 @@ async function codeAnalyzerBySwc(
     if (module.constructor.name !== 'NormalModule') {
       continue
     }
-    const originalSource = (module as webpack.NormalModule).originalSource()
+    const normalModule = module as webpack.NormalModule
+    if (!normalModule.type.startsWith('javascript')) {
+      // Only analyze JavaScript modules
+      continue
+    }
+    const originalSource = normalModule.originalSource()
     if (!originalSource) {
       continue
     }
Repository: ggml-org/llama.cpp
Pull request: llama : std::move llm_bigram_bpe from work_queue

This commit updates the retrieval of llm_bigram_bpe objects from work_queue.top() by using std::move.

The motivation for this is to avoid the copying of the std::string `text` member of the llm_bigram_bpe struct.



- [x] I have read the [contributing guidelines](https://github.com/ggerganov/llama.cpp/blob/master/CONTRIBUTING.md)
- Self-reported review complexity:
  - [x] Low
  - [ ] Medium
  - [ ] High


Changed files:
filename: src/llama-vocab.cpp
patch: @@ -321,6 +321,21 @@ struct llm_tokenizer_spm {
 
 // TODO: there are a lot of common parts between spm and bpe tokenizers, should be refactored and reused
 
+template<typename T, typename Container = std::vector<T>, typename Compare = std::less<typename Container::value_type>>
+class llama_priority_queue : public std::priority_queue<T, Container, Compare> {
+public:
+    using std::priority_queue<T, Container, Compare>::priority_queue;
+
+    T pop_move() {
+        T item = std::move(this->c.front());
+        std::pop_heap(this->c.begin(), this->c.end(), this->comp);
+        this->c.pop_back();
+        return item;
+    }
+
+    void pop() =  delete;
+};
+
 struct llm_bigram_bpe {
     struct comparator {
         bool operator()(const llm_bigram_bpe & l, const llm_bigram_bpe & r) const {
@@ -329,7 +344,7 @@ struct llm_bigram_bpe {
     };
 
     using queue_storage = std::vector<llm_bigram_bpe>;
-    using queue = std::priority_queue<llm_bigram_bpe, queue_storage, comparator>;
+    using queue = llama_priority_queue<llm_bigram_bpe, queue_storage, comparator>;
     llm_symbol::index left;
     llm_symbol::index right;
     std::string text;
@@ -520,8 +535,7 @@ struct llm_tokenizer_bpe {
 
             // build token(s)
             while (!work_queue.empty()) {
-                auto bigram = work_queue.top();
-                work_queue.pop();
+                auto bigram = work_queue.pop_move();
 
                 auto & left_symbol = symbols[bigram.left];
                 auto & right_symbol = symbols[bigram.right];
Repository: denoland/deno
Pull request: fix(ext/node): handle non-ws upgrade headers

Fixes https://github.com/denoland/deno/issues/27091

Don't perform a WS upgrade on non-ws upgrades by checking the value.


Changed files:
filename: ext/node/polyfills/http.ts
patch: @@ -548,8 +548,11 @@ class ClientRequest extends OutgoingMessage {
         incoming.statusMessage = res.statusText;
         incoming.upgrade = null;
 
-        for (const [key, _value] of res.headers) {
-          if (key.toLowerCase() === "upgrade") {
+        for (const [key, value] of res.headers) {
+          if (
+            key.toLowerCase() === "upgrade" &&
+            value.toLowerCase() === "websocket"
+          ) {
             incoming.upgrade = true;
             break;
           }

filename: tests/unit_node/http_test.ts
patch: @@ -372,6 +372,35 @@ Deno.test("[node/http] request default protocol", async () => {
   assertEquals(clientRes!.complete, true);
 });
 
+Deno.test("[node/http] request non-ws upgrade header", async () => {
+  const { promise, resolve } = Promise.withResolvers<void>();
+  const server = http.createServer((_req, res) => {
+    res.writeHead(200, { "upgrade": "h2,h2c" });
+    res.end("ok");
+  });
+  server.listen(() => {
+    const req = http.request(
+      {
+        host: "localhost",
+        // deno-lint-ignore no-explicit-any
+        port: (server.address() as any).port,
+      },
+      (res) => {
+        res.on("data", () => {});
+        res.on("end", () => {
+          server.close();
+        });
+        assertEquals(res.statusCode, 200);
+      },
+    );
+    req.end();
+  });
+  server.on("close", () => {
+    resolve();
+  });
+  await promise;
+});
+
 Deno.test("[node/http] request with headers", async () => {
   const { promise, resolve } = Promise.withResolvers<void>();
   const server = http.createServer((req, res) => {
Repository: facebook/react
Pull request: Turn off enableFormActions in Meta build

This is enabled in the canary channels, but because it's relatively untested, we'll disable it at Meta until they're ready to start trying it out. It can change some behavior even if you don't intentionally start using the API.

The reason it's not a dynamic flag is that it affects the external Fizz runtime, which currently can't read flags at runtime.

Changed files:
filename: packages/shared/forks/ReactFeatureFlags.www.js
patch: @@ -73,7 +73,7 @@ export const enableLegacyCache = true;
 export const enableCacheElement = true;
 export const enableFetchInstrumentation = false;
 
-export const enableFormActions = true;
+export const enableFormActions = false;
 
 export const disableJavaScriptURLs = true;
Repository: denoland/deno
Pull request: fix(ext/node): expose brotli stream APIs

Fixes https://github.com/denoland/deno/issues/27170

Changed files:
filename: ext/node/polyfills/_zlib.mjs
patch: @@ -654,4 +654,4 @@ util.inherits(DeflateRaw, Zlib);
 util.inherits(InflateRaw, Zlib);
 util.inherits(Unzip, Zlib);
 
-export { Deflate, DeflateRaw, Gunzip, Gzip, Inflate, InflateRaw, Unzip };
+export { Deflate, DeflateRaw, Gunzip, Gzip, Inflate, InflateRaw, Unzip, Zlib };

filename: ext/node/polyfills/zlib.ts
patch: @@ -33,8 +33,10 @@ import {
   unzipSync,
 } from "ext:deno_node/_zlib.mjs";
 import {
+  BrotliCompress,
   brotliCompress,
   brotliCompressSync,
+  BrotliDecompress,
   brotliDecompress,
   brotliDecompressSync,
   createBrotliCompress,
@@ -113,21 +115,7 @@ export class BrotliOptions {
     notImplemented("BrotliOptions.prototype.constructor");
   }
 }
-export class BrotliCompress {
-  constructor() {
-    notImplemented("BrotliCompress.prototype.constructor");
-  }
-}
-export class BrotliDecompress {
-  constructor() {
-    notImplemented("BrotliDecompress.prototype.constructor");
-  }
-}
-export class ZlibBase {
-  constructor() {
-    notImplemented("ZlibBase.prototype.constructor");
-  }
-}
+
 export { constants };
 
 export default {
@@ -216,12 +204,13 @@ export default {
   Z_SYNC_FLUSH: constants.Z_SYNC_FLUSH,
   Z_VERSION_ERROR: constants.Z_VERSION_ERROR,
   ZLIB_VERNUM: constants.ZLIB_VERNUM,
-  ZlibBase,
 };
 
 export {
+  BrotliCompress,
   brotliCompress,
   brotliCompressSync,
+  BrotliDecompress,
   brotliDecompress,
   brotliDecompressSync,
   codes,

filename: tests/unit_node/zlib_test.ts
patch: @@ -3,8 +3,10 @@
 import { assert, assertEquals, assertThrows } from "@std/assert";
 import { fromFileUrl, relative } from "@std/path";
 import {
+  BrotliCompress,
   brotliCompress,
   brotliCompressSync,
+  BrotliDecompress,
   brotliDecompress,
   brotliDecompressSync,
   constants,
@@ -237,3 +239,28 @@ Deno.test("crc32()", () => {
   // @ts-expect-error: passing an object
   assertThrows(() => crc32({}), TypeError);
 });
+
+Deno.test("BrotliCompress", async () => {
+  const deffered = Promise.withResolvers<void>();
+  // @ts-ignore: BrotliCompress is not typed
+  const brotliCompress = new BrotliCompress();
+  // @ts-ignore: BrotliDecompress is not typed
+  const brotliDecompress = new BrotliDecompress();
+
+  brotliCompress.pipe(brotliDecompress);
+
+  let data = "";
+  brotliDecompress.on("data", (v: Buffer) => {
+    data += v.toString();
+  });
+
+  brotliDecompress.on("end", () => {
+    deffered.resolve();
+  });
+
+  brotliCompress.write("hello");
+  brotliCompress.end();
+
+  await deffered.promise;
+  assertEquals(data, "hello");
+});
Repository: huggingface/transformers
Pull request: fix: add __bool__ operator to tokenizer to avoid bloated asserts

When a user does `assert tokenizer` to ensure that the tokenizer is not None, they inadvertently set off a rather expensive process in the `__len__()` operator. This fix adds a trivial `__bool__()` that returns True, so that a `None` tokenizer asserts and an actual tokenizer returns True when asserted, without calling length op.

A user can already address this by doing `assert tokenizer is not None`, but failing to do so is a big unexpected gotcha that shot me personally in the foot so I figured I'd at least propose a fix, which is trivial.

# What does this PR do?

This PR results in a roughly 300,000x speed-up when trying to `assert` a tokenizer object:

```python
import time
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("some-tokenizer")
start = time.perf_counter()
for i in range(4000):
    assert tokenizer
elapsed = time.perf_counter() - start
print(elapsed)
# PRE PR:  50.92993460899743
# POST PR:  0.0001564559934195131
```

## Before submitting
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker or @n1t0 might have opinions, otherwise not sure who might review this. It's very trivial.


Changed files:
filename: src/transformers/tokenization_utils_fast.py
patch: @@ -278,6 +278,12 @@ def get_added_vocab(self) -> dict[str, int]:
         """
         return {k.content: v for v, k in sorted(self.added_tokens_decoder.items(), key=lambda item: item[0])}
 
+    def __bool__(self) -> bool:
+        """
+        Returns True, to avoid expensive `assert tokenizer` gotchas.
+        """
+        return True
+
     def __len__(self) -> int:
         """
         Size of the full vocabulary with the added tokens.
Repository: electron/electron
Pull request: build: update appveyor image to latest version

This PR updates appveyor.yml to the latest baked image, e-133.0.6878.0.

Notes: none

Changed files:
filename: appveyor-woa.yml
patch: @@ -29,7 +29,7 @@
 
 version: 1.0.{build}
 build_cloud: electronhq-16-core
-image: e-133.0.6852.0
+image: e-133.0.6878.0
 environment:
   GIT_CACHE_PATH: C:\Users\appveyor\libcc_cache
   ELECTRON_OUT_DIR: Default

filename: appveyor.yml
patch: @@ -29,7 +29,7 @@
 
 version: 1.0.{build}
 build_cloud: electronhq-16-core
-image: e-133.0.6852.0
+image: e-133.0.6878.0
 environment:
   GIT_CACHE_PATH: C:\Users\appveyor\libcc_cache
   ELECTRON_OUT_DIR: Default
Repository: ggml-org/llama.cpp
Pull request: llama : fix typo in xcda_array_view comment [no ci]


- [x] I have read the [contributing guidelines](https://github.com/ggerganov/llama.cpp/blob/master/CONTRIBUTING.md)
- Self-reported review complexity:
  - [x] Low
  - [ ] Medium
  - [ ] High


Changed files:
filename: src/llama-vocab.cpp
patch: @@ -963,7 +963,7 @@ struct llm_tokenizer_ugm {
     /*
      * This structure is a view wrapper for XOR-compressed double array (XCDA)
      * See Shunsuke Kanda (2018). Space- and Time-Efficient String Dictionaries.
-     * Eeach bit-packed entry contains:
+     * Each bit-packed entry contains:
      * - BASE array value in bits 10-30
      * - LCHECK array value in bits 0-7
      * - LEAF array value in bit 9
Repository: facebook/react
Pull request: Go through the toString path for booleanish strings and .name property

This is consistent with what we used to do but not what we want to do.



Changed files:
filename: packages/react-dom-bindings/src/client/ReactDOMComponent.js
patch: @@ -685,7 +685,10 @@ function setProp(
         if (__DEV__) {
           checkAttributeStringCoercion(value, key);
         }
-        domElement.setAttribute(key, (value: any));
+        domElement.setAttribute(
+          key,
+          enableTrustedTypesIntegration ? (value: any) : '' + (value: any),
+        );
       } else {
         domElement.removeAttribute(key);
       }

filename: packages/react-dom-bindings/src/client/ReactDOMInput.js
patch: @@ -188,7 +188,7 @@ export function updateInput(
     if (__DEV__) {
       checkAttributeStringCoercion(name, 'name');
     }
-    node.name = name;
+    node.name = toString(getToStringValue(name));
   } else {
     node.removeAttribute('name');
   }
Repository: ggml-org/llama.cpp
Pull request: llava: add explicit instructions for llava-1.6

This commit contains a suggestion for the README.md in the llava example. The suggestion adds explicit instructions for how to convert a llava-1.6 model and run it using llava-cli.

The motivation for this is that having explicit instructions similar to the 1.5 instructions will make it easier for users to try this out.

Changed files:
filename: examples/llava/README.md
patch: @@ -59,14 +59,40 @@ python ./convert.py ../llava-v1.5-7b --skip-unknown
 Now both the LLaMA part and the image encoder is in the `llava-v1.5-7b` directory.
 
 ## LLaVA 1.6 gguf conversion
-
-1) Backup your pth/safetensor model files as llava-surgery modifies them
-2) Use `python llava-surgery-v2.py -C -m /path/to/hf-model` which also supports llava-1.5 variants pytorch as well as safetensor models:
+1) First clone a LLaVA 1.6 model:
+```console
+git clone https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b
+```
+2) Backup your pth/safetensor model files as llava-surgery modifies them
+3) Use `llava-surgery-v2.py` which also supports llava-1.5 variants pytorch as well as safetensor models:
+```console
+python examples/llava/llava-surgery-v2.py -C -m ../llava-v1.6-vicuna-7b/
+```
 - you will find a llava.projector and a llava.clip file in your model directory
-3) Copy the llava.clip file into a subdirectory (like vit), rename it to pytorch_model.bin and add a fitting vit configuration to the directory (https://huggingface.co/cmp-nct/llava-1.6-gguf/blob/main/config_vit.json) and rename it to config.json.
-4) Create the visual gguf model: `python ./examples/llava/convert-image-encoder-to-gguf.py -m ../path/to/vit --llava-projector ../path/to/llava.projector --output-dir ../path/to/output --clip-model-is-vision`
+4) Copy the llava.clip file into a subdirectory (like vit), rename it to pytorch_model.bin and add a fitting vit configuration to the directory:
+```console
+mkdir vit
+cp ../llava-v1.6-vicuna-7b/llava.clip vit/pytorch_model.bin
+cp ../llava-v1.6-vicuna-7b/llava.projector vit/
+curl -s -q https://huggingface.co/cmp-nct/llava-1.6-gguf/raw/main/config_vit.json -o vit/config.json
+```
+
+5) Create the visual gguf model:
+```console
+python ./examples/llava/convert-image-encoder-to-gguf.py -m vit --llava-projector vit/llava.projector --output-dir vit --clip-model-is-vision
+```
 - This is similar to llava-1.5, the difference is that we tell the encoder that we are working with the pure vision model part of CLIP
-5) Everything else as usual: convert.py the hf model, quantize as needed
+
+6) Then convert the model to gguf format:
+```console
+python ./convert.py ../llava-v1.6-vicuna-7b/
+```
+
+7) And finally we can run the llava-cli using the 1.6 model version:
+```console
+./llava-cli -m ../llava-v1.6-vicuna-7b/ggml-model-f16.gguf --mmproj vit/mmproj-model-f16.gguf --image some-image.jpg -c 4096
+```
+
 **note** llava-1.6 needs more context than llava-1.5, at least 3000 is needed (just run it at -c 4096)
 **note** llava-1.6 greatly benefits from batched prompt processing (defaults work)
Repository: facebook/react
Pull request: Update Flight fixture to use use() instead of Promise as a child

The Promise as a child case seems buggy. It ends up throwing the Promise as fatal when used in Sync rendering.

Changed files:
filename: fixtures/flight/src/index.js
patch: @@ -1,5 +1,5 @@
 import * as React from 'react';
-import {Suspense} from 'react';
+import {use, Suspense} from 'react';
 import ReactDOM from 'react-dom/client';
 import {createFromFetch, encodeReply} from 'react-server-dom-webpack/client';
 
@@ -27,7 +27,8 @@ let data = createFromFetch(
   }
 );
 
-// TODO: This transition shouldn't really be necessary but it is for now.
-React.startTransition(() => {
-  ReactDOM.hydrateRoot(document, data);
-});
+function Shell({data}) {
+  return use(data);
+}
+
+ReactDOM.hydrateRoot(document, <Shell data={data} />);