SixthSense 👁️

👉 Click here to try the Live Interactive Demo on Hugging Face Spaces

Privacy-conscious, on-device activity & productivity tracker that uses computer vision, audio classification, and LLM summarization to understand your work patterns — without streaming or cloud storage of raw data.

Architecture

Webcam Frame
    │
    ├─► YOLOv8 + ByteTrack ──► Object detection & persistent tracking
    ├─► MediaPipe Pose ────────► 33-landmark skeleton estimation
    ├─► Depth Anything V2 ─────► Monocular relative depth (foreground vs background)
    │
    └─► Rule-based State Machine ──► Activity label
              │                        (WORKING / USING PHONE / THINKING / PRESENT / AWAY)
              ▼
        SQLite + JSONL ──► Analytics Engine ──► Groq LLM Recap

Module	Role
periodic_logger.py	Main loop — captures snapshots, runs vision + audio pipeline, logs events
database.py	Dual-write storage (SQLite sixthsense.db + log.jsonl)
analytics.py	State smoothing, session segmentation, aggregate statistics
depth_estimator.py	Monocular depth estimation & desk-presence refinement
vlm_recapper.py	Groq Llama 3.3 daily natural-language recap generation

Setup

# 1. Clone & install dependencies
git clone https://github.com/avneetsingh7102/SixthSense.git && cd SixthSense
pip install -r requirements.txt

# 2. Download model weights (auto-downloaded on first run)
#    - yolov8n.pt            (YOLOv8 nano)
#    - pose_landmarker_full.task  (MediaPipe pose)

# 3. Configure Groq API key
cp .env.example .env
# Edit .env and set GROQ_API_KEY=your_key_here

Usage

# Run the main activity logger (opens webcam)
python periodic_logger.py

# Controls:
#   'c' → force an immediate snapshot
#   'q' → quit

# Run temporal analytics on logged data
python analytics.py

# Generate a daily LLM recap
python vlm_recapper.py

# Preview depth estimation standalone
python depth_estimator.py

Privacy

• No cloud streaming — all processing happens on-device.
• Periodic sampling — captures discrete snapshots at configurable intervals (default: 60s), not continuous video.
• Local storage only — all data stays in sixthsense.db and snapshots/.
• API key in .env — never committed to version control (.gitignore).

Technical Highlights

Decision	Rationale
ByteTrack over DeepSORT	Built into Ultralytics, lower latency, sufficient for single-user desk tracking
Monocular depth (relative, not metric)	No stereo camera needed; relative depth is sufficient to distinguish foreground user from background passerby
Rule-based state machine over LSTM	Deterministic, debuggable, and honest — labeled as "state smoothing" not "deep temporal modeling"
Groq (Llama 3.3 70B) for recaps	Fast inference, free tier available, avoids vendor lock-in
SQLite + JSONL dual-write	SQLite for structured queries; JSONL for easy streaming export and portability

License

MIT