Melati-AudioSet-EAT-Large

An AudioSet tagger: one model that names which of AudioSet's 527 sounds are in a clip, scoring 0.5123 mAP on AudioSet's evaluation set, ahead of every released single model we scored on the same clips (the best of the others, CED-base, 0.4997, and our own previous release (v3), 0.5105). It is EAT-large (309M parameters) distilled from the average of three released taggers (CED-base + EAT-large + SSLAM; 0.5166 averaged) on 730,841 AudioSet clips (2,011 hours), none of them from its evaluation set. Part of Melati, Scicom's speech-data pipeline.

model	mAP
this model	0.5123
previous release (v3)	0.5105
this model after stage 2	0.5059
this model after stage 1	0.5033
CED-base	0.4997	Dinkel et al., ICASSP 2024
EAT-large	0.4947	Chen et al., IJCAI 2024
SSLAM	0.4944	Alex et al., ICLR 2025
EAT-base	0.4901	Chen et al., IJCAI 2024
AST	0.4572	Gong et al., Interspeech 2021

All on the same 17,141 clips: AudioSet's evaluation set as agkphysics/AudioSet holds it today (YouTube removes clips over time, so every paper's copy differs and published numbers are not comparable across papers), each model in fp32 with its own preprocessing, mAP the mean over the 527 classes of their average precision.

Use

import torch, torchaudio, soundfile as sf
from transformers import AutoModel

model = AutoModel.from_pretrained("Scicom-intl/Melati-AudioSet-EAT-Large", trust_remote_code=True).eval()

wav, sr = sf.read("clip.wav", dtype="float32", always_2d=True)
wav = torch.from_numpy(wav.mean(1))
if sr != 16000:
    wav = torchaudio.functional.resample(wav, sr, 16000)
wav = wav - wav.mean()
fbank = torchaudio.compliance.kaldi.fbank(wav[None], htk_compat=True, sample_frequency=16000, use_energy=False,
                                          window_type="hanning", num_mel_bins=128, dither=0.0, frame_shift=10)
fbank = torch.nn.functional.pad(fbank, (0, 0, 0, max(0, 1024 - len(fbank))))[:1024]     # 10.24 s
x = ((fbank + 4.268) / (4.569 * 2))[None, None]                                          # [1, 1, 1024, 128]

with torch.no_grad():
    probs = torch.sigmoid(model(x))[0]
for i in probs.topk(5).indices.tolist():
    print(model.config.id2label[i], round(probs[i].item(), 3))

The input is EAT's: 16 kHz mono, 128-bin Kaldi filterbanks every 10 ms, 1024 frames (a longer clip in 10 s windows).

Labels

All 527 of AudioSet's classes, as model.config.id2label names them (the index is the output's column), under the 7 top-level categories of AudioSet's ontology (Google, CC BY-SA 4.0); the 29 classes it places under two categories are listed under the first. AP is this model's average precision on the class over the 17,141 evaluation clips in fp32, and clips is how many of them carry the label; the mAP above is the mean of the AP column.

Human sounds: 72 labels, mean AP 0.505

index	label	AP	clips
0	Speech	0.855	4,419
1	Male speech, man speaking	0.121	58
2	Female speech, woman speaking	0.159	59
3	Child speech, kid speaking	0.514	159
4	Conversation	0.482	50
5	Narration, monologue	0.151	55
6	Babbling	0.687	50
7	Speech synthesizer	0.538	54
8	Shout	0.235	52
9	Bellow	0.189	52
10	Whoop	0.269	54
11	Yell	0.295	45
12	Battle cry	0.882	54
13	Children shouting	0.609	53
14	Screaming	0.259	38
15	Whispering	0.845	51
16	Laughter	0.494	96
17	Baby laughter	0.688	56
18	Giggle	0.369	54
19	Snicker	0.307	54
20	Belly laugh	0.326	52
21	Chuckle, chortle	0.246	54
22	Crying, sobbing	0.572	40
23	Baby cry, infant cry	0.544	36
24	Whimper	0.429	45
25	Wail, moan	0.249	41
26	Sigh	0.413	46
27	Singing	0.369	292
28	Choir	0.721	84
29	Yodeling	0.673	53
30	Chant	0.498	48
31	Mantra	0.414	54
32	Male singing	0.283	55
33	Female singing	0.317	47
34	Child singing	0.653	50
35	Synthetic singing	0.609	45
36	Rapping	0.532	55
37	Humming	0.582	42
38	Groan	0.086	43
39	Grunt	0.437	43
40	Whistling	0.782	46
41	Breathing	0.186	49
42	Wheeze	0.371	51
43	Snoring	0.890	51
44	Gasp	0.313	47
45	Pant	0.515	47
46	Snort	0.263	50
47	Cough	0.623	42
48	Throat clearing	0.612	50
49	Sneeze	0.594	37
50	Sniff	0.430	52
51	Run	0.400	50
52	Shuffle	0.568	56
53	Walk, footsteps	0.504	49
54	Chewing, mastication	0.606	42
55	Biting	0.386	54
56	Gargling	0.704	41
57	Stomach rumble	0.844	41
58	Burping, eructation	0.736	48
59	Hiccup	0.786	53
60	Fart	0.835	48
61	Hands	0.552	49
62	Finger snapping	0.588	47
63	Clapping	0.553	52
64	Heart sounds, heartbeat	0.640	57
65	Heart murmur	0.389	49
66	Cheering	0.560	52
67	Applause	0.623	51
68	Chatter	0.765	46
69	Crowd	0.745	126
70	Hubbub, speech noise, speech babble	0.375	50
71	Children playing	0.714	48

Animal: 65 labels, mean AP 0.654

index	label	AP	clips
72	Animal	0.677	586
73	Domestic animals, pets	0.702	295
74	Dog	0.777	193
75	Bark	0.466	51
76	Yip	0.372	54
77	Howl	0.719	51
78	Bow-wow	0.521	62
79	Growling	0.413	56
80	Whimper (dog)	0.265	48
81	Cat	0.820	96
82	Purr	0.850	56
83	Meow	0.730	56
84	Hiss	0.450	78
85	Caterwaul	0.769	56
86	Livestock, farm animals, working animals	0.789	99
87	Horse	0.626	73
88	Clip-clop	0.528	55
89	Neigh, whinny	0.889	44
90	Cattle, bovinae	0.960	53
91	Moo	0.952	51
92	Cowbell	0.782	57
93	Pig	0.703	45
94	Oink	0.438	48
95	Goat	0.645	48
96	Bleat	0.530	49
97	Sheep	0.595	52
98	Fowl	0.895	179
99	Chicken, rooster	0.822	62
100	Cluck	0.678	42
101	Crowing, cock-a-doodle-doo	0.921	45
102	Turkey	0.950	56
103	Gobble	0.948	56
104	Duck	0.504	50
105	Quack	0.680	55
106	Goose	0.817	61
107	Honk	0.753	53
108	Wild animals	0.370	42
109	Roaring cats (lions, tigers)	0.537	44
110	Roar	0.648	50
111	Bird	0.493	187
112	Bird vocalization, bird call, bird song	0.354	151
113	Chirp, tweet	0.154	62
114	Squawk	0.363	50
115	Pigeon, dove	0.834	50
116	Coo	0.864	50
117	Crow	0.724	59
118	Caw	0.730	57
119	Owl	0.840	54
120	Hoot	0.965	53
121	Bird flight, flapping wings	0.767	46
122	Canidae, dogs, wolves	0.291	52
123	Rodents, rats, mice	0.489	51
124	Mouse	0.191	51
125	Patter	0.440	56
126	Insect	0.873	141
127	Cricket	0.674	55
128	Mosquito	0.811	47
129	Fly, housefly	0.885	94
130	Buzz	0.126	52
131	Bee, wasp, etc.	0.835	50
132	Frog	0.948	50
133	Croak	0.943	49
134	Snake	0.567	37
135	Rattle	0.079	52
136	Whale vocalization	0.748	48

Music: 146 labels, mean AP 0.488

index	label	AP	clips
137	Music	0.877	4,654
138	Musical instrument	0.442	369
139	Plucked string instrument	0.553	232
140	Guitar	0.705	274
141	Electric guitar	0.617	115
142	Bass guitar	0.537	52
143	Acoustic guitar	0.526	84
144	Steel guitar, slide guitar	0.376	56
145	Tapping (guitar technique)	0.515	54
146	Strum	0.292	56
147	Banjo	0.812	57
148	Sitar	0.523	49
149	Mandolin	0.494	55
150	Zither	0.663	49
151	Ukulele	0.614	53
152	Keyboard (musical)	0.533	134
153	Piano	0.417	83
154	Electric piano	0.452	55
155	Organ	0.838	115
156	Electronic organ	0.499	59
157	Hammond organ	0.609	57
158	Synthesizer	0.251	49
159	Sampler	0.182	54
160	Harpsichord	0.831	56
161	Percussion	0.636	215
162	Drum kit	0.654	73
163	Drum machine	0.309	60
164	Drum	0.577	171
165	Snare drum	0.660	72
166	Rimshot	0.666	69
167	Drum roll	0.651	56
168	Bass drum	0.639	81
169	Timpani	0.803	51
170	Tabla	0.673	54
171	Cymbal	0.605	60
172	Hi-hat	0.669	53
173	Wood block	0.703	52
174	Tambourine	0.735	54
175	Rattle (instrument)	0.441	49
176	Maraca	0.387	48
177	Gong	0.775	51
178	Tubular bells	0.352	53
179	Mallet percussion	0.325	46
180	Marimba, xylophone	0.408	57
181	Glockenspiel	0.411	48
182	Vibraphone	0.519	54
183	Steelpan	0.818	53
184	Orchestra	0.393	53
185	Brass instrument	0.817	179
186	French horn	0.796	51
187	Trumpet	0.625	50
188	Trombone	0.655	53
189	Bowed string instrument	0.587	101
190	String section	0.549	54
191	Violin, fiddle	0.421	53
192	Pizzicato	0.734	115
193	Cello	0.671	59
194	Double bass	0.709	52
195	Wind instrument, woodwind instrument	0.803	224
196	Flute	0.625	46
197	Saxophone	0.690	53
198	Clarinet	0.664	55
199	Harp	0.785	50
200	Bell	0.263	52
201	Church bell	0.723	54
202	Jingle bell	0.672	50
203	Bicycle bell	0.789	50
204	Tuning fork	0.770	53
205	Chime	0.774	49
206	Wind chime	0.836	47
207	Change ringing (campanology)	0.927	58
208	Harmonica	0.929	50
209	Accordion	0.875	55
210	Bagpipes	0.967	58
211	Didgeridoo	0.909	54
212	Shofar	0.896	52
213	Theremin	0.571	51
214	Singing bowl	0.658	51
215	Scratching (performance technique)	0.579	56
216	Pop music	0.229	70
217	Hip hop music	0.331	51
218	Beatboxing	0.656	46
219	Rock music	0.212	81
220	Heavy metal	0.308	60
221	Punk rock	0.259	58
222	Grunge	0.276	51
223	Progressive rock	0.373	64
224	Rock and roll	0.366	75
225	Psychedelic rock	0.248	51
226	Rhythm and blues	0.140	52
227	Soul music	0.262	56
228	Reggae	0.408	45
229	Country	0.275	56
230	Swing music	0.382	50
231	Bluegrass	0.521	60
232	Funk	0.249	49
233	Folk music	0.148	47
234	Middle Eastern music	0.271	43
235	Jazz	0.108	53
236	Disco	0.314	51
237	Classical music	0.119	51
238	Opera	0.592	54
239	Electronic music	0.292	52
240	House music	0.317	50
241	Techno	0.517	78
242	Dubstep	0.495	83
243	Drum and bass	0.381	66
244	Electronica	0.368	70
245	Electronic dance music	0.529	72
246	Ambient music	0.325	49
247	Trance music	0.393	52
248	Music of Latin America	0.260	49
249	Salsa music	0.637	53
250	Flamenco	0.465	54
251	Blues	0.259	55
252	Music for children	0.267	50
253	New-age music	0.305	52
254	Vocal music	0.286	66
255	A capella	0.554	54
256	Music of Africa	0.366	63
257	Afrobeat	0.462	51
258	Christian music	0.159	36
259	Gospel music	0.462	46
260	Music of Asia	0.180	50
261	Carnatic music	0.623	56
262	Music of Bollywood	0.473	36
263	Ska	0.428	54
264	Traditional music	0.254	54
265	Independent music	0.180	56
266	Song	0.128	135
267	Background music	0.351	54
268	Theme music	0.304	53
269	Jingle (music)	0.176	50
270	Soundtrack music	0.081	48
271	Lullaby	0.476	50
272	Video game music	0.500	47
273	Christmas music	0.105	44
274	Dance music	0.291	71
275	Wedding music	0.134	53
276	Happy music	0.226	53
277	Funny music	0.134	45
278	Sad music	0.319	47
279	Tender music	0.130	55
280	Exciting music	0.105	47
281	Angry music	0.762	53
282	Scary music	0.448	48

Natural sounds: 17 labels, mean AP 0.504

index	label	AP	clips
283	Wind	0.525	170
284	Rustling leaves	0.207	54
285	Wind noise (microphone)	0.557	139
286	Thunderstorm	0.803	53
287	Thunder	0.802	52
288	Water	0.583	239
289	Rain	0.766	86
290	Raindrop	0.581	54
291	Rain on surface	0.627	52
292	Stream	0.454	52
293	Waterfall	0.607	53
294	Ocean	0.504	59
295	Waves, surf	0.451	54
296	Steam	0.470	55
297	Gurgling	0.162	52
298	Fire	0.413	54
299	Crackle	0.065	50

Sounds of things: 166 labels, mean AP 0.539

index	label	AP	clips
300	Vehicle	0.580	698
301	Boat, Water vehicle	0.685	223
302	Sailboat, sailing ship	0.603	57
303	Rowboat, canoe, kayak	0.604	58
304	Motorboat, speedboat	0.467	56
305	Ship	0.410	56
306	Motor vehicle (road)	0.143	77
307	Car	0.437	250
308	Vehicle horn, car horn, honking	0.477	54
309	Toot	0.576	53
310	Car alarm	0.756	50
311	Power windows, electric windows	0.712	54
312	Skidding	0.646	49
313	Tire squeal	0.302	50
314	Car passing by	0.434	53
315	Race car, auto racing	0.446	64
316	Truck	0.387	89
317	Air brake	0.404	56
318	Air horn, truck horn	0.694	51
319	Reversing beeps	0.644	51
320	Ice cream truck, ice cream van	0.833	52
321	Bus	0.499	54
322	Emergency vehicle	0.956	121
323	Police car (siren)	0.798	52
324	Ambulance (siren)	0.784	54
325	Fire engine, fire truck (siren)	0.898	51
326	Motorcycle	0.314	50
327	Traffic noise, roadway noise	0.150	61
328	Rail transport	0.838	161
329	Train	0.835	160
330	Train whistle	0.516	56
331	Train horn	0.675	54
332	Railroad car, train wagon	0.840	155
333	Train wheels squealing	0.879	56
334	Subway, metro, underground	0.763	58
335	Aircraft	0.622	99
336	Aircraft engine	0.676	51
337	Jet engine	0.712	58
338	Propeller, airscrew	0.277	55
339	Helicopter	0.635	52
340	Fixed-wing aircraft, airplane	0.558	59
341	Bicycle	0.374	56
342	Skateboard	0.828	55
343	Engine	0.429	173
344	Light engine (high frequency)	0.589	53
345	Dental drill, dentist's drill	0.376	52
346	Lawn mower	0.594	56
347	Chainsaw	0.640	52
348	Medium engine (mid frequency)	0.113	53
349	Heavy engine (low frequency)	0.129	56
350	Engine knocking	0.268	56
351	Engine starting	0.454	56
352	Idling	0.218	80
353	Accelerating, revving, vroom	0.283	131
354	Door	0.262	57
355	Doorbell	0.617	49
356	Ding-dong	0.265	42
357	Sliding door	0.320	54
358	Slam	0.243	51
359	Knock	0.628	45
360	Tap	0.508	55
361	Squeak	0.457	51
362	Cupboard open or close	0.399	48
363	Drawer open or close	0.503	54
364	Dishes, pots, and pans	0.340	51
365	Cutlery, silverware	0.451	51
366	Chopping (food)	0.686	47
367	Frying (food)	0.834	58
368	Microwave oven	0.412	52
369	Blender	0.821	51
370	Water tap, faucet	0.444	66
371	Sink (filling or washing)	0.598	52
372	Bathtub (filling or washing)	0.619	42
373	Hair dryer	0.893	50
374	Toilet flush	0.865	46
375	Toothbrush	0.144	50
376	Electric toothbrush	0.409	49
377	Vacuum cleaner	0.774	47
378	Zipper (clothing)	0.702	52
379	Keys jangling	0.370	52
380	Coin (dropping)	0.571	55
381	Scissors	0.515	52
382	Electric shaver, electric razor	0.553	49
383	Shuffling cards	0.816	57
384	Typing	0.647	52
385	Typewriter	0.674	53
386	Computer keyboard	0.712	51
387	Writing	0.573	51
388	Alarm	0.324	53
389	Telephone	0.347	45
390	Telephone bell ringing	0.707	47
391	Ringtone	0.606	46
392	Telephone dialing, DTMF	0.688	47
393	Dial tone	0.628	50
394	Busy signal	0.659	46
395	Alarm clock	0.487	54
396	Siren	0.921	181
397	Civil defense siren	0.923	52
398	Buzzer	0.379	55
399	Smoke detector, smoke alarm	0.772	48
400	Fire alarm	0.738	50
401	Foghorn	0.502	54
402	Whistle	0.458	49
403	Steam whistle	0.740	66
404	Mechanisms	0.367	91
405	Ratchet, pawl	0.363	48
406	Clock	0.502	54
407	Tick	0.224	55
408	Tick-tock	0.453	56
409	Gears	0.373	47
410	Pulleys	0.419	51
411	Sewing machine	0.885	56
412	Mechanical fan	0.430	55
413	Air conditioning	0.332	55
414	Cash register	0.859	53
415	Printer	0.838	50
416	Camera	0.539	51
417	Single-lens reflex camera	0.631	51
418	Tools	0.432	93
419	Hammer	0.631	58
420	Jackhammer	0.615	55
421	Sawing	0.540	55
422	Filing (rasp)	0.728	56
423	Sanding	0.879	55
424	Power tool	0.421	72
425	Drill	0.581	55
426	Explosion	0.277	46
427	Gunshot, gunfire	0.792	138
428	Machine gun	0.784	49
429	Fusillade	0.613	44
430	Artillery fire	0.813	48
431	Cap gun	0.885	45
432	Fireworks	0.755	48
433	Firecracker	0.633	42
434	Burst, pop	0.052	52
435	Eruption	0.418	52
436	Boom	0.506	53
437	Wood	0.514	151
438	Chop	0.511	49
439	Splinter	0.495	54
440	Crack	0.229	46
441	Glass	0.403	54
442	Chink, clink	0.180	48
443	Shatter	0.345	47
444	Liquid	0.137	53
445	Splash, splatter	0.493	42
446	Slosh	0.355	48
447	Squish	0.094	53
448	Drip	0.327	52
449	Pour	0.499	49
450	Trickle, dribble	0.194	50
451	Gush	0.357	50
452	Fill (with liquid)	0.206	53
453	Spray	0.592	54
454	Pump (liquid)	0.419	53
455	Stir	0.816	57
456	Boiling	0.321	55
457	Sonar	0.661	48
458	Arrow	0.784	55
459	Whoosh, swoosh, swish	0.426	51
460	Thump, thud	0.293	46
461	Thunk	0.379	47
462	Electronic tuner	0.684	53
463	Effects unit	0.653	102
464	Chorus effect	0.605	52
465	Basketball bounce	0.686	52

Source-ambiguous sounds: 40 labels, mean AP 0.389

index	label	AP	clips
466	Bang	0.139	51
467	Slap, smack	0.733	50
468	Whack, thwack	0.292	46
469	Smash, crash	0.493	50
470	Breaking	0.480	47
471	Bouncing	0.197	52
472	Whip	0.767	50
473	Flap	0.152	51
474	Scratch	0.512	53
475	Scrape	0.043	47
476	Rub	0.791	127
477	Roll	0.360	50
478	Crushing	0.344	54
479	Crumpling, crinkling	0.531	51
480	Tearing	0.406	49
481	Beep, bleep	0.388	49
482	Ping	0.448	51
483	Ding	0.357	54
484	Clang	0.379	50
485	Squeal	0.240	51
486	Creak	0.178	50
487	Rustle	0.089	51
488	Whir	0.148	54
489	Clatter	0.149	58
490	Sizzle	0.758	64
491	Clicking	0.702	48
492	Clickety-clack	0.598	57
493	Rumble	0.401	50
494	Plop	0.847	45
495	Jingle, tinkle	0.273	48
496	Hum	0.396	71
497	Zing	0.400	43
498	Boing	0.809	40
499	Crunch	0.507	46
500	Silence	0.062	47
501	Sine wave	0.605	53
502	Harmonic	0.068	50
503	Chirp tone	0.340	52
504	Sound effect	0.064	40
505	Pulse	0.135	48

Channel, environment and background: 21 labels, mean AP 0.301

index	label	AP	clips
506	Inside, small room	0.220	479
507	Inside, large room or hall	0.130	133
508	Inside, public space	0.174	45
509	Outside, urban or manmade	0.125	177
510	Outside, rural or natural	0.073	157
511	Reverberation	0.329	55
512	Echo	0.385	53
513	Noise	0.113	56
514	Environmental noise	0.561	48
515	Static	0.554	50
516	Mains hum	0.600	56
517	Distortion	0.543	107
518	Sidetone	0.236	53
519	Cacophony	0.180	53
520	White noise	0.117	54
521	Pink noise	0.393	50
522	Throbbing	0.356	52
523	Vibration	0.173	56
524	Television	0.332	49
525	Radio	0.426	52
526	Field recording	0.291	55

How it was trained

The student is trained to the average of the teachers' probabilities (binary cross-entropy against soft targets; AudioSet's own labels are not used) on AudioSet's strongly labelled training clips (102,123, every event timed by hand, from CLAPv2/audioset_strong), its balanced set (11,397) and 617,321 clips of its unbalanced set, any clip of the evaluation set left out, in 3 stages:

1. From EAT-large fine-tuned on AudioSet-2M (0.4942 on these clips in bf16), 10 epochs on the strongly labelled set and the balanced set (113,520 clips, 311 hours), learning rate 1e-5: 0.5027 at its best (bf16).
2. From stage 1, 3 epochs on the strongly labelled set, the balanced set and 617,321 clips of the unbalanced set (730,841 clips, 2,011 hours), learning rate 1e-5: 0.5054 at its best (bf16).
3. From stage 2, 5 epochs on the strongly labelled set and the balanced set (113,520 clips, 311 hours), learning rate 1e-5: 0.5118 at its best (bf16).

The previous release (v3) trained stage 3 with a second head beside the first that learns the hand-timed events of the 102,123 strongly labelled clips (binary cross-entropy every 160 ms): 0.5105 on these clips in fp32. Trained again with that loss off, the stage reaches 0.5123, so the timed events did not lift the clip head, and this release leaves them out; its lift over stage 2 (0.5059) comes from the rest of the stage, another learning-rate cycle on the strongly labelled set and the balanced set without mixup. Every stage: SpecAugment, warm-up and cosine decay, bf16 with gradient checkpointing, and an exponential moving average of the weights (0.9995) whose best evaluation is kept; stages 1 and 2 also mixed half their batches (mixup). The code is audioset/ in Melati.

Limitations

• One evaluation set: AudioSet's own. On speech-heavy material (podcasts) it tags what AudioSet tags, which is not a measure of how well it times or places an event inside a long recording.
• Clip-level: a probability for each class over the whole input. The frame head trained beside the previous release (v3) was not released either: on AudioSet's 16,896 strongly labelled test clips it places sounds in time less well than PretrainedSED (frame mAP 0.4102 against 0.4261 on the 364 classes both have), which is what Melati's own pipeline uses for that.
• The training clips are 730,841 of AudioSet's 2 million; more of its unbalanced set is the obvious lever.

Licence

MIT, as EAT's code and weights it starts from. The teachers it learned from are released under Apache-2.0 (CED-base) and MIT (EAT-large, SSLAM). AudioSet's labels are CC BY 4.0 (Google); its audio is YouTube's and is not redistributed here.

Citation

@misc{zolkepli2026melati,
  title  = {Melati: Multilingual Expressive speech, Labelled for Affect, Tags and Interjections},
  author = {Husein Zolkepli},
  year   = {2026},
  note   = {Scicom (MSC) Berhad}
}

We thank Scicom (MSC) Berhad for the compute.