Skill: openlark/mediapipe (version 1.0.0)

--- name: mediapipe description: on-device ML pipeline framework for vision, text, audio, and LLM inference. Cross-platform deployment to Android, iOS, web, desktop, edge devices, and IoT. --- # Google MediaPipe ## Overview MediaPipe is Google's open-source framework for building on-device machine learning pipelines. It provides cross-platform APIs for vision, text, audio, and LLM inference tasks, plus a low-level graph-based pipeline framework for custom ML workloads. ## Covers - Computer vision tasks (face detection, face mesh, hand tracking, pose estimation, holistic landmarks, object detection, image classification/segmentation, gesture recognition) - Text tasks (text classification, text embedding, language detection) - Audio classification, (4) On-device LLM inference with MediaPipe GenAI/Tasks - Model customization with MediaPipe Model Maker - Visualizing/building ML pipelines with the MediaPipe Framework (graphs, calculators, packets) - Drawing/rendering landmarks and detection results onto images/video - Understanding MediaPipe's architecture and component ecosystem (Solutions, Tasks, Model Maker, Studio, Framework) ### Architecture MediaPipe has two layers: 1. **MediaPipe Solutions** (high-level) — Pre-built, ready-to-use ML tasks via cross-platform APIs. Use these for most applications. 2. **MediaPipe Framework** (low-level) — Graph-based pipeline builder (packets, graphs, calculators) for custom on-device ML pipelines. C++ core with Android/iOS bindings. The Solutions layer consists of: - **MediaPipe Tasks** — Cross-platform libraries (Python, Android, iOS, Web/JS, C++) wrapping pre-trained models - **MediaPipe Models** — Pre-trained TFLite model bundles downloadable per task - **MediaPipe Model Maker** — Fine-tune/customize models with your own data - **MediaPipe Studio** — Browser-based no-code benchmarking and prototyping tool ## Installation ### Python ```bash pip install mediapipe ``` Latest version as of 2026-05: `0.10.35`. The Python package bundles all tasks. Models are downloaded separately at runtime or pre-downloaded. ### Android Add to `build.gradle`: ``` implementation 'com.google.mediapipe:tasks-vision:0.10.35' ``` Replace `vision` with `text`, `audio`, or `genai` as needed. ### Web / JavaScript ```bash npm install @mediapipe/tasks-vision ``` Available packages: `@mediapipe/tasks-vision`, `@mediapipe/tasks-text`, `@mediapipe/tasks-audio`, `@mediapipe/tasks-genai`. ### iOS (CocoaPods) ``` pod 'MediaPipeTasksVision' ``` ## Quick Start — Python Examples ### Face Detection ```python import mediapipe as mp from mediapipe.tasks import python from mediapipe.tasks.python import vision model_path = '/absolute/path/to/blaze_face_short_range.tflite' base_options = python.BaseOptions(model_asset_path=model_path) options = vision.FaceDetectorOptions(base_options=base_options) detector = vision.FaceDetector.create_from_options(options) image = mp.Image.create_from_file('photo.jpg') result = detector.detect(image) for detection in result.detections: bbox = detection.bounding_box print(f"Face at x={bbox.origin_x}, y={bbox.origin_y}, " f"w={bbox.width}, h={bbox.height}, " f"score={detection.categories[0].score}") ``` ### Hand Landmark Detection ```python model_path = '/path/to/hand_landmarker.task' options = vision.HandLandmarkerOptions( base_options=python.BaseOptions(model_asset_path=model_path), num_hands=2) detector = vision.HandLandmarker.create_from_options(options) image = mp.Image.create_from_file('hands.jpg') result = detector.detect(image) for hand_landmarks in result.hand_landmarks: for lm in hand_landmarks: print(f"Landmark: x={lm.x}, y={lm.y}, z={lm.z}") ``` ### Pose Landmark Detection ```python model_path = '/path/to/pose_landmarker_lite.task' options = vision.PoseLandmarkerOptions( base_options=python.BaseOptions(model_asset_path=model_path)) detector = vision.PoseLandmarker.create_from_options(options) image = mp.Image.create_from_file('person.jpg') result = detector.detect(image) # result.pose_landmarks is a list of NormalizedLandmark lists (33 landmarks each) # result.pose_world_landmarks provides 3D world coordinates ``` ### Object Detection ```python model_path = '/path/to/efficientdet_lite0.tflite' options = vision.ObjectDetectorOptions( base_options=python.BaseOptions(model_asset_path=model_path), max_results=5) detector = vision.ObjectDetector.create_from_options(options) image = mp.Image.create_from_file('scene.jpg') result = detector.detect(image) for detection in result.detections: print(f"Class: {detection.categories[0].category_name}, " f"BBox: {detection.bounding_box}") ``` ### Text Classification ```python from mediapipe.tasks.python import text model_path = '/path/to/text_classifier.tflite' options = text.TextClassifierOptions( base_options=python.BaseOptions(model_asset_path=model_path)) classifier = text.TextClassifier.create_from_options(options) result = classifier.classify("I absolutely loved this movie!") for category in result.classifications[0].categories: print(f"{category.category_name}: {category.score:.4f}") ``` ### Drawing Landmarks on Images ```python import cv2 import mediapipe as mp from mediapipe.tasks.python import vision from mediapipe.framework.formats import landmark_pb2 # ... detect landmarks ... # Convert result landmarks to NormalizedLandmarkList hand_landmarks_proto = landmark_pb2.NormalizedLandmarkList() hand_landmarks_proto.landmark.extend([ landmark_pb2.NormalizedLandmark(x=lm.x, y=lm.y, z=lm.z) for lm in result.hand_landmarks[0] ]) # Draw on image annotated = mp.solutions.drawing_utils.draw_landmarks( image_rgb, hand_landmarks_proto, mp.solutions.hands.HAND_CONNECTIONS, mp.solutions.drawing_styles.get_default_hand_landmarks_style(), mp.solutions.drawing_styles.get_default_hand_connections_style() ) ``` ## Vision Tasks All vision tasks support three running modes: `IMAGE`, `VIDEO`, and `LIVE_STREAM`. ### FaceDetector - **Models**: `blaze_face_short_range.tflite` (2m), `blaze_face_full_range.tflite` (5m) - **Output**: bounding boxes with 6 keypoints (eyes, nose, mouth, ears) - **Options**: `min_detection_confidence`, `min_suppression_threshold` ### FaceLandmarker - **Models**: `face_landmarker.task` (478 3D landmarks), `face_landmarker_v2_with_blendshapes.task` - **Output**: 478 face mesh landmarks, 52 blendshape scores, face transformation matrix - **Options**: `num_faces`, `min_face_detection_confidence`, `min_tracking_confidence`, `output_face_blendshapes`, [REDACTED_SECRET] ### HandLandmarker - **Models**: `hand_landmarker.task` - **Output**: 21 hand landmarks per hand, handedness classification (left/right), world landmarks - **Options**: `num_hands`, `min_hand_detection_confidence`, `min_tracking_confidence` ### PoseLandmarker - **Models**: `pose_landmarker_lite.task`, `pose_landmarker_full.task`, `pose_landmarker_heavy.task` - **Output**: 33 body pose landmarks, world 3D landmarks, segmentation mask - **Options**: `num_poses`, `min_pose_detection_confidence`, `min_tracking_confidence`, `output_segmentations` ### HolisticLandmarker - **Models**: `holistic_landmarker.task` - **Output**: Combined face (478), pose (33), and hand (21×2) landmarks simultaneously - **Options**: `min_face_detection_confidence`, `min_pose_detection_confidence`, `min_hand_landmarks_confidence`, `output_face_blendshapes` ### GestureRecognizer - **Models**: `gesture_recognizer.task` - **Output**: Predefined gesture categories from hand landmarks (e.g., "Thumb_Up", "Victory", "Closed_Fist", "Open_Palm", "Pointing_Up", "ILoveYou") - **Options**: `min_hand_detection_confidence`, `min_tracking_confidence`, [REDACTED_SECRET] ### ObjectDetector - **Models**: `efficientdet_lite0.tflite` through `efficientdet_lite2.tflite` (COCO 80 classes) - **Output**: Bounding boxes with category labels and scores - **Options**: `max_results`, `score_threshold`, `category_allowlist`, `category_denylist` ### ImageClassifier - **Models**: `efficientnet_lite0.tflite` through `efficientnet_lite4.tflite` (ImageNet 1k) - **Output**: Classification category list with scores - **Options**: `max_results`, `score_threshold`, `category_allowlist/denylist` ### ImageEmbedder - **Models**: `mobilenet_v3_small.tflite`, `mobilenet_v3_large.tflite` - **Output**: Feature embedding vectors (float or quantized) for similarity/comparison - **Options**: `l2_normalize`, `quantize` ### ImageSegmenter - **Models**: Various segmentation models (DeepLab, selfie segmenter, hair segmenter) - **Output**: Category mask and/or confidence mask - **Options**: `output_category_mask`, `output_confidence_masks` ### InteractiveSegmenter - **Models**: `magic_touch.tflite`, `sam.tflite` - **Output**: Segmentation mask for a user-specified region of interest (click/tap) - **Options**: `output_category_mask`, `output_confidence_masks` ## Text Tasks ### TextClassifier - **Models**: `text_classifier.tflite` (BERT-based), custom models via Model Maker - **Output**: Classification categories with scores (sentiment, topic, etc.) - **Options**: `max_results`, `score_threshold`, `category_allowlist/denylist` ### TextEmbedder - **Models**: `universal_sentence_encoder.tflite`, `bert_embedder.tflite` - **Output**: Text embedding vectors for semantic similarity, clustering, retrieval - **Options**: `l2_normalize`, `quantize` ### LanguageDetector - **Models**: `language_detector.tflite` - **Output**: Detected language BCP-47 code(s) with probabilities (supports 110+ languages) ## Audio Tasks ### AudioClassifier - **Models**: `yamnet.tflite` (521 audio event classes), custom models - **Output**: Audio event classification with timestamps - **Input**: Audio clips (mono, 16kHz sample rate) or streaming audio buffers - **Options**: `max_results`, `score_threshold`, `category_allowlist/denylist` - Supports `AUDIO_CLIPS` and `AUDIO_STREAM` running modes ## LLM Inference (GenAI) MediaPipe includes on-device LLM inference via MediaPipe Tasks GenAI (as of v0.10.35): - **JavaScript**: `@mediapipe/tasks-genai` package for web-based LLM inference - **Android**: Tasks GenAI for NPU-accelerated on-device LLM - **Python**: LLM converter utilities for blockwise int4 quantization, weight compression - Supports configurable quantization policies and supervised round quantization (SRQ) ## MediaPipe Model Maker Customize pre-trained models with your own data without ML expertise: ```bash pip install mediapipe-model-maker ``` ```python from mediapipe_model_maker import text_classifier data = text_classifier.Dataset.from_csv('reviews.csv') model = text_classifier.create(data) model.export_model() ``` Supports customization for text classification, object detection, image classification, and gesture recognition. Model Maker uses transfer learning with a few hundred examples. ## MediaPipe Framework (Low-Level) For building custom on-device ML pipelines beyond pre-built solutions: ### Core Concepts - **Packets** — Typed data containers (images, tensors, landmarks) that flow through the graph - **Graphs** — Directed acyclic graphs of calculator nodes defining the pipeline topology - **Calculators** — Processing nodes that consume input packets and produce output packets - **Streams** — Named data pathways connecting calculator inputs/outputs - **Side Packets** — Configuration data injected at graph initialization ### Graph Configuration (.pbtxt) ```protobuf input_stream: "input_video" output_stream: "output_video" node { calculator: "ImageToTensorCalculator" input_stream: "IMAGE:input_video" output_stream: "TENSORS:image_tensor" } node { calculator: "InferenceCalculator" input_stream: "TENSORS:image_tensor" output_stream: "TENSORS:detection_tensors" options { [mediapipe.InferenceCalculatorOptions.ext] { model_path: "/path/to/model.tflite" } } } ``` ### Supported Platforms for Framework - C++ (Bazel build system) - Android (AAR, JNI bindings) - iOS (framework) - Desktop (Linux, macOS, Windows via C++) The Framework is **not** available for Python or web — use MediaPipe Tasks/Solutions for those platforms. ## Model Management ### Downloading Models Models are hosted at `https://storage.googleapis.com/mediapipe-models/`. Download programmatically: ```python # Python: download helper (if available) or manual curl # wget https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/latest/hand_landmarker.task ``` For production, download models ahead of time and bundle with your app. ### Model Path Requirements - **Python**: Must use absolute paths for `model_asset_path`. Relative paths or `pathlib.Path` objects may fail. - **Web**: Pass Wasm file URLs; must be served from same origin or with CORS headers. - **Android**: Place `.task`/`.tflite` files in `src/main/assets/`. ## Common Patterns & Best Practices ### Running Modes ```python # IMAGE mode — single image inference options = vision.FaceDetectorOptions( base_options=python.BaseOptions(model_asset_path=model_path), running_mode=vision.RunningMode.IMAGE) # VIDEO mode — frame sequence with timestamps options = vision.FaceDetectorOptions( base_options=python.BaseOptions(model_asset_path=model_path), running_mode=vision.RunningMode.VIDEO) # result = detector.detect_for_video(image, timestamp_ms) # LIVE_STREAM mode — async callback-based for camera streams def on_result(result, image, timestamp): pass # handle result asynchronously options = vision.FaceDetectorOptions( base_options=python.BaseOptions(model_asset_path=model_path), running_mode=vision.RunningMode.LIVE_STREAM, result_callback=on_result) ``` ### Lazy Resource Cleanup All task objects implement context manager protocol: ```python with vision.FaceDetector.create_from_options(options) as detector: result = detector.detect(image) # detector is automatically closed ``` ### Image Handling ```python import mediapipe as mp # From file image = mp.Image.create_from_file('photo.jpg') # From numpy array (must be RGB, uint8) import cv2 cv_image = cv2.imread('photo.jpg') rgb_image = cv2.cvtColor(cv_image, cv2.COLOR_BGR2RGB) mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb_image) ``` ### GPU Delegation ```python base_options = python.BaseOptions( model_asset_path=model_path, delegate=python.BaseOptions.Delegate.GPU) # or .CPU (default) ``` ### Error Handling ```python try: detector = vision.FaceDetector.create_from_options(options) except Exception as e: print(f"Failed to create detector: {e}") # Common issues: wrong model path, incompatible model version, missing TFLite runtime ``` ## Model Versions & Compatibility - Model `.task` bundles (new format) vs legacy `.tflite` + metadata - Task bundle encapsulates model + metadata + pre/post processing configs - Always check model/task version compatibility — model versions are tied to specific MediaPipe SDK versions - Download latest models from the official model hub ## Legacy Notes - Legacy solutions (pre-2023) are deprecated as of March 2023 — use Tasks API instead - Old `mp.solutions.hands`, `mp.solutions.pose`, `mp.solutions.face_mesh` APIs are legacy - The new Tasks API (`mediapipe.tasks.python.vision.HandLandmarker`) supersedes legacy APIs - Legacy code still in `mediapipe.solutions.*` namespace; use Tasks for new projects - The Framework layer continues to be maintained for custom pipeline development ## Key Links - **Official Docs**: https://developers.google.com/mediapipe - **GitHub**: https://github.com/google-ai-edge/mediapipe - **Samples**: https://github.com/google-ai-edge/mediapipe-samples - **Model Hub**: https://developers.google.com/mediapipe/solutions/models - **Studio**: https://mediapipe-studio.web.app - **PyPI**: https://pypi.org/project/mediapipe/ - **Model Maker**: https://developers.google.com/mediapipe/solutions/model_maker - **Paper**: https://arxiv.org/abs/1906.08172


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)