citizen-cyberscience-centre/pybossa — Issue #53: Create Table Transcription Platform in PyBossa
Author: lucasmation
Created: 2012-02-15 12:54:03+00:00
State: open

following emails, work in Berlin and in Campina Grande and agregating all the information here:

taks: create a Table-Transcription platoform for extracting data from tables in scanned pages. This would be a general platform, but one use that interest me is to extract historical data on demography, heath and crime in XIX Brazil.

The idea: half the page with an image and half the page with a some sort of spreadsheet where the data is transcribed by the volunteer. There may be intermediate steps before the table appears in which the volunteer identifies if there is a table in the image and fill in some metadata about the table (title, stating and ending page (in case of tables spanning more than one page, type of data, source).

Other infos:

1)Campina Grande folks (@nigini and Jey) already developed a platform. Details:
-Task management is done using the original Bossa (PHP)
- The table is done by a jQuery Spreadsheet plugin (http://www.visop-dev.com/Project+jQuery.sheet) (@nigini: "Basically, we get the text inside spreadsheet cells, convert it to JSON and store at the "TaskRun" instance. When another task for the same page will start, we recover the data from the last TaskRun and push it back at the spreadsheet. In this way we have a Wiki alike collaboration, and the systems maintains all data versions"(.
You can test it by:
- create acount at: http://bacalhau.lsd.ufcg.edu.br/table_transcriber/create_account_form.php?next_url=home.php
- login and start transcribing at: bacalhau.lsd.ufcg.edu.br/table_transcriber/bossa_apps.php
- the code behind that application is at: http://svn.lsd.ufcg.edu.br/repos/sc/table_transcriber > user: "anonymous" , leave password blank.
- the project documentation is maitained here: http://redmine.lsd.ufcg.edu.br/projects/datadigitizer (but it is out of date and the server may be down).

2) We need to think what is the best table-spread sheet solution for this. Options that I know of:
- jQuery Spreadsheet . The advantage is that is all runs in our server. But it may be dificult to ajust stuff and less intuitive to the user.
- googleDcos spreadsheet. We can use "googleDocs API 3.0" (pytho) to manipulate table creation, user access, etc. And "googleSpreadsheet API" to manipulate how the page is embeded and etc. Disadvantage: other server, relies on google. Advantage: more powerfull, more intuitive.

@rufus, @nigini, @teleyinex : do you know any othe options? In your opinion, what is the best option?

3) I have some ideas on how tasks should be atributed, which probably should be incorporated into PyBossa structure. Basically there are two ways people can colaborate:
- sequentially: with one volunteer building on the work of a previous volunteer. This is how a "wiki" works.
- in parallel: with the same task beeing done in parallel by many volunteers and the results compared for consistency.
In the case of the table transcription we need both. Given a page (i) we can define a sej (j) of sequetitial translations (t).
So a Job is described by: JOBi,j,t.
Professional and academic table transcriptions using humans always use two parallel transcriptions, which are compared for consistency.
Lets put some indexes as this could help in coding. So in this case, for translating page i=1, we would have:

Sequence j=1: JOB i=1,j=1,t=1 > JOB i=1,j=1,t=2 > JOB i=1,j=1,t=3 > ... > JOB i=1,j=1,t=n1 (when a user declares the transcription to be over.
Sequence j=2: JOB i=1,j=2,t=1 > JOB i=1,j=2,t=2 > JOB i=1,j=2,t=3 > ... > JOB i=1,j=2,t=n2

we would then have to compare (authomatically or by experienced users or project supervisors) the final iterations of each sequence: JOB i=1,j=1,t=n1 and JOB i=1,j=2,t=n2

@teleyinex , @rgrp : is this flexible sequential and/or parallel way of atributing tasks already build into PyBossa?