Graphics Interface 2014 Session #5 “Video and Collaboration”

VisionSketch:

Integrated Support for
Example-Centric Programming of
Image Processing Applications
Jun Kato1,2, Takeo Igarashi1
1The University of Tokyo
2National Institute of Advanced Industrial Science and Technology

Background

Cameras are ubiquitous
Surveillance camera image quoted from http://en.wikipedia.org/wiki/File:Three_Surveillance_cameras.jpg under CC BY-SA 3.0

Surveillance

Time-lapse photography

Interesting events Interesting information
can be detected can be extracted2

Background

Programming is needed
Monitoring 24h/7days?

Computers never get tired

Regarding the variety of desired information,using preset programs is not enough.
3

Problem
Background

Programming is difficult• Programming in general is to create abstract logic• IDEs are equipped with textual interfaces• Tuning algorithms takes long time by iterative cycleof changing code and restarting the program4

Motivation

Programming should be easier• Programming in general is to create abstract logic
Start off by choosing a concrete example• IDEs are equipped with textual interfaces
Get graphical feedback with help of the example• Tuning algorithms takes long time by iterative cycleof changing code and restarting the program
Update the program and get immediate feedback

Example-Centric Programming
5Our Approach

Integrated Support forDemo
Example-Centric Programming
Target apps: image processing applications (fixed viewpoint)Goal:address difficulties of current mainstream IDEs
by allowing programmers to…
• Start off by choosing a concrete example• Get graphical feedback with help of the example• Update the program and get immediate feedback

Method:provide three interlinked interfaces

CanvasVisual Editor

Code Editor
6

Related Work

Subtext[Edwards, OOPSLA Onward ‘04]

Integrated support for
example-centricprogramming• allows to write
incomplete code
• as well as
concrete test code
Designed for simple character-based applications
No graphical representations
7

Related WorkConMan[Haeberli, SIGGRAPH ‘88]

VPL, casual program execution using recorded data

Designed for tuning parameters of CG rendering
Mere visualization, no graphical editing
8

Related Work

DejaVu[Kato et al., UIST ‘12]

Canvas interface forvisualizing contentsof any variables• during execution• after the execution

Designed for record & replay of program executions
No support for direct manipulation of graphics
9

User Experience

Preliminary user study• To collect user feedback and investigate
applications and limitations
Participants: • 5 male programmers with professional
programming experience, aged 23-36• 4 of them have used OpenCV for static
image processing
Procedures: • Pre-study questionnaire• Work on a video (selected based on
their interest) to create their own app• Post-study questionnaire and interview

Purpose:10

User Experience

Example applications

Door Watcher
Disc Jockey Analyzer
Good-for-eatingSensor
11

User Experience

Observations from the studyIDE + user code = application• Normally: toolkit + user code = application• Suitable for prototyping (programmer = user)
Many simple components > a few complex code
• When computational cost doesn’t matter…
• Preference for graphical operations over coding
Improvements on code editor needed• Criticism on not providing graphical feedbacks• Combination with past work (e.g. DejaVu) desired
12

Conclusion

Limitations and Future Work
Technical limitations• Images are assumed to be captured from static
viewpoints but the system can be extended tohandle dynamic viewpoints.
• Graphs with loops are not supported but can besupported.
Intrinsic limitation• Example-centric approach cannot be applied tobuilding apps with real-time feedback loops.
13

Conclusion

Integrated Support for
Example-Centric Programming
Proposed and evaluated design of VisionSketch IDEwith three interlinked interfaces to aid examplecentric programming of image processing apps.
Open-source distribution for Windows & Mac OS X
http://junkato.jp/visionsketch/
CanvasFor sketching
program overview

Visual Editor

Code Editor

For drawing shapes toFor writing code tochoose & tune algorithms implement algorithms
14

Appendix

15

VisionSketch

CanvasGraphical user interface forgraphical overview
Start off by choosing a concrete example• Program execution casuallycontrolled by the slider
Get graphical feedback with help of the example• Typical visual programming language but withgraphical representations for all components• Freeform comments sketched on the canvas
16

VisionSketch

Visual Editor
Graphical user interface forchoosing image processing
component and tuning its parameters
Start off by choosing a concrete example• Parameter-based code completion (drawing shapesnarrows down the list of applicable components)
Update the program and get immediate feedback• Interactive graphical feedback of processing results
17

VisionSketch

Code Editor
Text-based code editor forediting and creating image
processing components
Update the program and get immediate feedback• Seamless switch between text and visual interface• Selective updates of corresponding components
without restarting the whole program

18