Repository: Davidyz/VectorCode

Issue:
[BUG] Treesitter doesn't handle correctly when `chunk_size = -1`
**Describe the bug**
I was just trying this out on a rails project. It was able to index html.erb files, css, js, and configuration, but it seems like ruby does not work. I was looking for language support, it seems like anything tree sitter supports should be supported, and i use tree sitter ruby every day :) 

**To Reproduce**
Steps to reproduce the behavior:
1. echo "class Foobar; end" > foobar.rb
2. vectorcode vectorise foobar.rb
3. notice the zeros
4. vectorcode query foobar => Empty collection!

**Expected behavior**
Would expect it to work with ruby

**Platform information:**
 - If the issue is about the CLI, attach the output of `pipx runpip vectorcode freeze`:
```
annotated-types==0.7.0
anyio==4.9.0
asgiref==3.8.1
backoff==2.2.1
bcrypt==4.3.0
build==1.2.2.post1
cachetools==5.5.2
certifi==2025.1.31
charset-normalizer==3.4.1
chroma-hnswlib==0.7.6
chromadb==0.6.3
click==8.1.8
coloredlogs==15.0.1
Deprecated==1.2.18
distro==1.9.0
durationpy==0.9
fastapi==0.115.11
filelock==3.18.0
flatbuffers==25.2.10
fsspec==2025.3.0
google-auth==2.38.0
googleapis-common-protos==1.69.2
grpcio==1.71.0
h11==0.14.0
httpcore==1.0.7
httptools==0.6.4
httpx==0.28.1
huggingface-hub==0.29.3
humanfriendly==10.0
idna==3.10
importlib_metadata==8.6.1
importlib_resources==6.5.2
Jinja2==3.1.6
joblib==1.4.2
kubernetes==32.0.1
markdown-it-py==3.0.0
MarkupSafe==3.0.2
mdurl==0.1.2
mmh3==5.1.0
monotonic==1.6
mpmath==1.3.0
networkx==3.4.2
numpy==2.2.4
oauthlib==3.2.2
onnxruntime==1.21.0
opentelemetry-api==1.31.0
opentelemetry-exporter-otlp-proto-common==1.31.0
opentelemetry-exporter-otlp-proto-grpc==1.31.0
opentelemetry-instrumentation==0.52b0
opentelemetry-instrumentation-asgi==0.52b0
opentelemetry-instrumentation-fastapi==0.52b0
opentelemetry-proto==1.31.0
opentelemetry-sdk==1.31.0
opentelemetry-semantic-conventions==0.52b0
opentelemetry-util-http==0.52b0
orjson==3.10.15
overrides==7.7.0
packaging==24.2
pathspec==0.12.1
pillow==11.1.0
posthog==3.21.0
protobuf==5.29.4
psutil==7.0.0
pyasn1==0.6.1
pyasn1_modules==0.4.1
pydantic==2.10.6
pydantic_core==2.27.2
Pygments==2.19.1
PyPika==0.48.9
pyproject_hooks==1.2.0
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
PyYAML==6.0.2
regex==2024.11.6
requests==2.32.3
requests-oauthlib==2.0.0
rich==13.9.4
rsa==4.9
safetensors==0.5.3
scikit-learn==1.6.1
scipy==1.15.2
sentence-transformers==3.4.1
setuptools==77.0.1
shellingham==1.5.4
shtab==1.7.1
six==1.17.0
sniffio==1.3.1
starlette==0.46.1
sympy==1.13.1
tabulate==0.9.0
tenacity==9.0.0
threadpoolctl==3.6.0
tokenizers==0.21.1
torch==2.6.0
tqdm==4.67.1
transformers==4.49.0
tree-sitter==0.24.0
tree-sitter-c-sharp==0.23.1
tree-sitter-embedded-template==0.23.2
tree-sitter-language-pack==0.6.1
tree-sitter-yaml==0.7.0
typer==0.15.2
typing_extensions==4.12.2
urllib3==2.3.0
uvicorn==0.34.0
uvloop==0.21.0
VectorCode==0.4.13
watchfiles==1.0.4
websocket-client==1.8.0
websockets==15.0.1
wrapt==1.17.2
zipp==3.21.0
```


**System Information:**

> For Mac users, please also mention whether you're using intel or apple silicon devices.
on an m4 macbook air

Discussion hints:
codecov-commenter: ## Welcome to [Codecov](https://codecov.io?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Davidyz) :tada:

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered :open_umbrella:

Solution patch:
diff --git a/src/vectorcode/chunking.py b/src/vectorcode/chunking.py
index 40bbfad..58c5d46 100644
--- a/src/vectorcode/chunking.py
+++ b/src/vectorcode/chunking.py
@@ -111,6 +111,9 @@ class TreeSitterChunker(ChunkerBase):
         assert os.path.isfile(data)
         with open(data) as fin:
             content = fin.read()
+        if self.__chunk_size < 0:
+            yield content
+            return
         parser = None
         lexer = self.__guess_type(data, content)
         if lexer is not None:


Test patch:
diff --git a/tests/test_chunking.py b/tests/test_chunking.py
index 9f1eb20..8267c1e 100644
--- a/tests/test_chunking.py
+++ b/tests/test_chunking.py
@@ -8,6 +8,8 @@ class TestChunking:
     file_chunker = FileChunker()
 
     def test_string_chunker(self):
+        string_chunker = StringChunker(chunk_size=-1, overlap_ratio=0.5)
+        assert list(string_chunker.chunk("hello world")) == ["hello world"]
         string_chunker = StringChunker(chunk_size=5, overlap_ratio=0.5)
         assert list(string_chunker.chunk("hello world")) == [
             "hello",
@@ -36,6 +38,7 @@ class TestChunking:
         file_path = __file__
         ratio = 0.5
         chunk_size = 100
+
         with open(file_path) as fin:
             string_chunker = StringChunker(chunk_size=chunk_size, overlap_ratio=ratio)
             string_chunks = list(string_chunker.chunk(fin.read()))
@@ -51,11 +54,10 @@ class TestChunking:
             assert string_chunk == file_chunk
 
 
-def test_treesitter_chunker():
+def test_treesitter_chunker_python():
     """Test TreeSitterChunker with a sample file using tempfile."""
     chunker = TreeSitterChunker(chunk_size=30)
 
-    # test python
     test_content = r"""
 def foo():
     return "foo"
@@ -72,7 +74,9 @@ def bar():
     assert chunks == ['def foo():\n    return "foo"', 'def bar():\n    return "bar"']
     os.remove(test_file)
 
-    # test lua
+
+def test_treesitter_chunker_lua():
+    chunker = TreeSitterChunker(chunk_size=30)
     test_content = r"""
 function foo()
   return "foo"
@@ -93,6 +97,50 @@ end
     os.remove(test_file)
 
 
+def test_treesitter_chunker_ruby():
+    chunker = TreeSitterChunker(chunk_size=30)
+    test_content = r"""
+def greet_user(name)
+  "Hello, #{name.capitalize}!"
+end
+
+def add_numbers(a, b)
+  a + b
+end
+    """
+
+    with tempfile.NamedTemporaryFile(mode="w", delete=False, suffix=".rb") as tmp_file:
+        tmp_file.write(test_content)
+        test_file = tmp_file.name
+
+    chunks = list(chunker.chunk(test_file))
+    assert len(chunks) > 0
+
+    os.remove(test_file)
+
+
+def test_treesitter_chunker_neg_chunksize():
+    chunker = TreeSitterChunker(chunk_size=-1)
+    test_content = r"""
+def greet_user(name)
+  "Hello, #{name.capitalize}!"
+end
+
+def add_numbers(a, b)
+  a + b
+end
+    """
+
+    with tempfile.NamedTemporaryFile(mode="w", delete=False, suffix=".rb") as tmp_file:
+        tmp_file.write(test_content)
+        test_file = tmp_file.name
+
+    chunks = list(chunker.chunk(test_file))
+    assert len(chunks) == 1
+
+    os.remove(test_file)
+
+
 def test_treesitter_chunker_fallback():
     """Test that TreeSitterChunker falls back to StringChunker when no parser is found."""
     chunk_size = 30