Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,92 @@
|
|
1 |
-
---
|
2 |
-
license: apache-2.0
|
3 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: apache-2.0
|
3 |
+
---
|
4 |
+
|
5 |
+
|
6 |
+
# embedfile
|
7 |
+
|
8 |
+
|
9 |
+
Experimental CLI tool for generating and searching text embeddings, built on
|
10 |
+
[llamafile](https://github.com/Mozilla-Ocho/llamafile),
|
11 |
+
[`sqlite-vec`](https://github.com/asg017/sqlite-vec),
|
12 |
+
[`sqlite-lembed`](https://github.com/asg017/sqlite-lembed),
|
13 |
+
[the SQLite CLI](https://www.sqlite.org/cli.html),
|
14 |
+
and a few other SQLite extensions.
|
15 |
+
|
16 |
+
|
17 |
+
| Model | embedfile | Size (f16 quant) |
|
18 |
+
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
|
19 |
+
| [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | [`all-MiniLM-L6-v2.f16.embedfile`](https://huggingface.co/asg017/embedfile/resolve/main/all-MiniLM-L6-v2.f16.embedfile) | `56MB` |
|
20 |
+
| [mixedbread-ai/mxbai-embed-xsmall-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-xsmall-v1) | [`mxbai-embed-xsmall-v1-f16.embedfile`](https://huggingface.co/asg017/embedfile/resolve/main/mxbai-embed-xsmall-v1-f16.embedfile) | `61MB` |
|
21 |
+
| [nomic-ai/nomic-embed-text-v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) | [`nomic-embed-text-v1.5.f16.embedfile`](https://huggingface.co/asg017/embedfile/resolve/main/nomic-embed-text-v1.5.f16.embedfile) | `273MB` |
|
22 |
+
| [snowflake-arctic-embed-m-v1.5](https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5) | [`snowflake-arctic-embed-m-v1.5-f16.embedfile`](https://huggingface.co/asg017/embedfile/resolve/main/snowflake-arctic-embed-m-v1.5-f16.embedfile) | `221MB` |
|
23 |
+
| - | [`embedfile`](https://huggingface.co/asg017/embedfile/resolve/main/embedfile) (no embedded model) | `12MB` |
|
24 |
+
|
25 |
+
|
26 |
+
embedfiles run on Linux, Mac, and Windows computers in the same file, thanks to [cosmopolitan](https://github.com/jart/cosmopolitan).
|
27 |
+
You can embed data from CSVs, JSON, NDJSON, and txt files from the CLI, or "eject" to the `sqlite3` CLI at any time.
|
28 |
+
|
29 |
+
|
30 |
+
Here's an example, using MixedBread's xsmall model:
|
31 |
+
|
32 |
+
```
|
33 |
+
$ wget https://huggingface.co/asg017/embedfile/resolve/main/mxbai-embed-xsmall-v1-f16.embedfile
|
34 |
+
$ chmod u+x mxbai-embed-xsmall-v1-f16.embedfile
|
35 |
+
$ ./mxbai-embed-xsmall-v1-f16.embedfile --version
|
36 |
+
embedfile 0.0.1-alpha.1, llamafile 0.8.16, SQLite 3.47.0, sqlite-vec=v0.1.6, sqlite-lembed=v0.0.1-alpha.8
|
37 |
+
|
38 |
+
```
|
39 |
+
|
40 |
+
This executable file already has `sqlite-vec`, `sqlite-lembed`, and the embeddings model pre-configured. Test that embeddings work with:
|
41 |
+
|
42 |
+
|
43 |
+
```
|
44 |
+
./mxbai-embed-xsmall-v1-f16.embedfile embed 'hello!'
|
45 |
+
[-0.058174,0.043776,0.030660,...]
|
46 |
+
```
|
47 |
+
|
48 |
+
You can embed data from CSV, JSON, NDJSON, and .txt files and save the results to a SQLite database. Here we are embedding the `text` column in the `dbpedia.min.csv` file, outputting to a `dbpedia.db` database.
|
49 |
+
|
50 |
+
```
|
51 |
+
$ ./mxbai-embed-xsmall-v1-f16.embedfile import --embed text dbpedia.min.csv dbpedia.db
|
52 |
+
INSERT INTO vec_items SELECT rowid, lembed("text") FROM temp.source;
|
53 |
+
100%|ββββββββββββββββββββ| 10000/10000 [02:00<00:00, 83/s]
|
54 |
+
β dbpedia.min.csv imported into dbpedia.db, 10000 items
|
55 |
+
```
|
56 |
+
|
57 |
+
That was 10,000 rows with 820,604 tokens. I got 83 embeddings per second on my older 2019 Intel Macbook. On my M1 Mac Mini I get 173 embbedings/second, and I'm sure it's faster on newer macs.
|
58 |
+
|
59 |
+
Once indexed, you can search with the `search` command:
|
60 |
+
|
61 |
+
```
|
62 |
+
$ ./mxbai-embed-xsmall-v1-f16.embedfile search dbpedia.db 'global warming'
|
63 |
+
3240 0.852299 Attribution of recent climate change is the effort to scientifically ascertain mechanisms ...
|
64 |
+
6697 0.904844 The global warming controversy concerns the public debate over whether global warming is occurring, how ...
|
65 |
+
...
|
66 |
+
```
|
67 |
+
|
68 |
+
|
69 |
+
At any point, if you want to "eject" and run SQL scripts yourself, the `sh` command will fire up the `sqlite3` CLI with all extensions and embeddings models pre-configured.
|
70 |
+
|
71 |
+
```
|
72 |
+
$ ./mxbai-embed-xsmall-v1-f16.embedfile sh
|
73 |
+
SQLite version 3.47.0 2024-10-21 16:30:22
|
74 |
+
Enter ".help" for usage hints.
|
75 |
+
Connected to a transient in-memory database.
|
76 |
+
Use ".open FILENAME" to reopen on a persistent database.
|
77 |
+
sqlite> .mode qbox
|
78 |
+
sqlite> select sqlite_version(), vec_version(), lembed_version();
|
79 |
+
ββββββββββββββββββββ¬ββββββββββββββββ¬βββββββββββββββββββ
|
80 |
+
β sqlite_version() β vec_version() β lembed_version() β
|
81 |
+
βββοΏ½οΏ½οΏ½ββββββββββββββββΌββββββββββββββββΌβββββββββββββββββββ€
|
82 |
+
β '3.47.0' β 'v0.1.6' β 'v0.0.1-alpha.8' β
|
83 |
+
ββββββββββββββββββββ΄ββββββββββββββββ΄βββββββββββββββββββ
|
84 |
+
sqlite> select vec_to_json(vec_slice(lembed('hello!'), 0, 8)) as sample;
|
85 |
+
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
86 |
+
β sample β
|
87 |
+
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
|
88 |
+
β '[-0.058174,0.043776,0.030660,0.047412,-0.059377,-0.036267,0 β
|
89 |
+
β .038117,0.005184]' β
|
90 |
+
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
91 |
+
```
|
92 |
+
|