npm package: @hg0428/diverse-lemmas-data
Description: Language data files for @hg0428/diverse-lemmas. 50+ languages.
Author: hg0428
Latest version: 1.0.0

# diverse-lemmas

**Offline multilingual lemmatization for browser and Node.js**

50+ languages • On-demand downloads • Local caching • Zero runtime dependencies

## Features

- 🌍 **50+ languages** - Hebrew, Korean, Spanish, German, Russian, and many more
- 📦 **Tiny core package** - Just 15KB, language data downloaded on demand
- 💾 **Smart caching** - IndexedDB (browser) or filesystem (Node.js)
- ⚡ **Fast lookups** - Sub-microsecond after initial load
- 🔌 **Works everywhere** - Browser, Node.js, Deno, Bun

## Installation

```bash
npm install @hg0428/diverse-lemmas
```

## Quick Start

```javascript
import { loadLanguage, lemmatize, LANGUAGES } from '@hg0428/diverse-lemmas';

// Load Spanish (downloads ~15MB on first use, then cached)
const es = await loadLanguage('es');

// Lemmatize words
es.lemmatizeWord('hablando');
// => { lemmas: ['hablar'], method: 'direct' }

es.lemmatizeWord('niños');
// => { lemmas: ['niño'], method: 'direct' }

// Or use the quick API
const result = await lemmatize('es', 'corriendo');
// => { lemmas: ['correr'], method: 'direct' }
```

## API

### Loading Languages

```javascript
import { loadLanguage, loadLanguages, getLemmatizer } from '@hg0428/diverse-lemmas';

// Load a single language
const de = await loadLanguage('de');

// Load multiple languages in parallel
const langs = await loadLanguages(['en', 'es', 'fr']);
// => Map { 'en' => Lemmatizer, 'es' => Lemmatizer, 'fr' => Lemmatizer }

// Get already-loaded lemmatizer (null if not loaded)
const cached = getLemmatizer('de');
```

### Lemmatization

```javascript
// Single word
lemmatizer.lemmatizeWord('running');
// => { lemmas: ['run'], method: 'direct' }

// Multiple words
lemmatizer.lemmatizeWords(['dogs', 'running', 'quickly']);
// => [{ word: 'dogs', lemmas: ['dog'], method: 'direct' }, ...]

// Full text (auto-tokenizes)
lemmatizer.lemmatizeText('The dogs were running quickly');
// => [{ word: 'The', lemmas: ['the'], method: 'direct' }, ...]
```

### Cache Management

```javascript
import { 
  getCachedLanguages, 
  isLanguageCached, 
  removeLanguage, 
  clearCache,
  getCacheInfo 
} from '@hg0428/diverse-lemmas';

// List cached languages
await getCachedLanguages();
// => ['en', 'es', 'de']

// Check if cached
await isLanguageCached('fr');
// => false

// Get cache info
await getCacheInfo('es');
// => { cachedAt: 1704067200000, sizeKB: 15360 }

// Remove from cache
await removeLanguage('de');

// Clear everything
await clearCache();
```

### Language Info

```javascript
import { LANGUAGES, getSupportedLanguages, getLanguageInfo } from '@hg0428/diverse-lemmas';

// All supported languages
getSupportedLanguages();
// => ['he', 'ko', 'es', 'fr', 'it', 'en', 'de', ...]

// Get language metadata
getLanguageInfo('es');
// => { name: 'Spanish', source: 'simplemma', hasAmbiguity: true, sizeKB: 15360 }

// Large languages (>50MB) - consider warning users
LANGUAGES.fi.large  // => true (Finnish: 97MB)
LANGUAGES.pl.large  // => true (Polish: 97MB)
LANGUAGES.sw.large  // => true (Swahili: 111MB)
```

### Configuration

```javascript
import { setCDN, getCDN } from '@hg0428/diverse-lemmas';

// Use a custom CDN or self-hosted files
setCDN('https://my-cdn.com/lemmas');

// Get current CDN
getCDN();
// => 'https://my-cdn.com/lemmas'
```

## Supported Languages

| Category | Languages |
|----------|-----------|
| **Full support** (with ambiguity) | Hebrew, Korean, Spanish, French, Italian |
| **Major European** | English, German, Portuguese, Dutch, Russian, Polish, Swedish |
| **Nordic** | Danish, Norwegian (Bokmål/Nynorsk), Finnish, Icelandic |
| **Eastern European** | Czech, Slovak, Hungarian, Romanian, Bulgarian, Ukrainian |
| **Celtic** | Irish, Welsh, Scottish Gaelic |
| **Other** | Turkish, Indonesian, Latin, and 20+ more |

See [LANGUAGES.md](./LANGUAGES.md) for the complete list with sizes.

## How It Works

1. **Core package** (`@hg0428/diverse-lemmas`) is tiny (~15KB)
2. **Language files** are hosted on CDN (jsDelivr by default)
3. **On first use**, the language is downloaded and cached locally
4. **Subsequent uses** load instantly from local cache
5. **Browser**: Uses IndexedDB for persistent storage
6. **Node.js**: Uses filesystem (`~/.diverse-lemmas/cache/`)

## Data Sources

- **Simplemma** - 49 languages from [simplemma](https://github.com/adbar/simplemma)
- **Hebrew** - Custom dictionary + Stanza + verb conjugations
- **Korean** - Universal Dependencies Korean-GSD and Korean-Kaist

## License

MIT