if a http.Head request fails, issue a Get request. Some server do not like or support Head requests
part 1 for AN003
Squashed 'analyser/swagger-ui/' changes from 5794d1c..cd2c75e

cd2c75e expected 6 SQL fields, but got only 5 - fix SQL statement
ec209a0 add full url to the check message to get more helpful debugging text
43e252d AN001 was erroneous by SQL, fix it
4a1a142 SQL query optimization. Now takes 1/4 of the time
2feea30 deleted datasets need not get checked
c400e1d new API function
b4b2217 fix typo in when rendering the OGDAT-metadata specification
087fde1 add a new route
2c906ae Merge branch 'master' of github.com:the42/ogdat
41f07f1 Fix NOT NULL definition
a5d60be common place for redis keys
a87d0a2 new functionality: Add DateSets based on their taxonomy membership
2075511 redis.AddFlat is meaningless as it uses fmt.Print(#v)
e7a43f8 new version of go-restful handles flat / gzip encoding in the Container
e54864e Keep serving the API even the analysing functionality is disabled
ec3a4e3 slightly rebuild the logic how taxonomies are stored in redis and retrieved via the API
d3e72db add documentation about the currently naive approach to rebuild statistics upon data change
1258fbd Execute the http server in its own go-routine
db43409 remove debugging code
6cbe221 enable XHR CORS OPTIONS
9183ff0 test why a channel doesn't get called
a3f39dd an API base path may be empty
6ca1f9b Get API base path from environemnt variable
aa4d0b5 for configuration, no local port required
a26f38a rename environment variable
3cb7834 changes due to update of go-restful
48f3d0a read hostname from environemnt variable
7b5148d prepare for public availability
6cefbd9 add check results to redis database
b235cb1 there is no need for this code any longer since the UI lives in its own repository
2af8b33 delete user interface component. It is now available as github.com/the42/ogdat-analyserui
e5e40fc Merge branch 'master' of github.com:the42/ogdat
ebc47f1 Merge branch 'master' of github.com:the42/ogdat
5135448 DRY - use an anoynmous function to prevent repetition in a select case
756bd41 commit 2/2: no versioning in API urls
0dd0ffd commit 1/2: no versioning in API urls
5be5a38 Service is available with an api version specifier
3ff337b Merge commit '6c0dd8b39114c746c4e4c0280a554787922b91aa'
6c0dd8b Squashed 'analyser/swagger-ui/' changes from dd3037f..5794d1c
4d066ce Beginnings of the actual analyse-UI
9106d37 new url for dynamic documentation
92e2c56 prepare for heroku
dbaf019 CORS-enable the API
03f9c05 instead of panicking on a fatal error, set the appropriate response code
2a3edfb gzip-encode the response
e0eadea in case of error, quit the listen event loop
6a5b0d8 guarantee that redis keys are truly unique
d037a8e make retrieval of sorted sets generic
0be4006 make retrieval of sorted sets generic
3442e30 convert redis connection handling to using a pool
1115d54 changes, enhancements and refactoring
64281c9 rename file
f6fa6db In our case, git submodules are not a good idea, use subtrees instead.
16b3ae6 Add 'analyser/swagger-ui/' from commit 'dd3037f481979ec15b8f1c19154d800565622a53'
dca1d7c submodules are not the way to go
913d6b6 no need for a closure, cf. https://github.com/emicklei/go-restful/issues/21#issuecomment-20252820
2170f48 First stab at the restful API for Angular frontend
2d47536 add swagger-ui submodule
67d711d Move code arround. No new functionality or additions.
97b1228 Publish a data change event into Redis
2266729 add Redis Publish helper function
6399b39 in case of error, return error, not nil
6f9eff9 analyser: Populate geographic information into Redis
2bafb0e change sql schema to reflect changes
86a114e ogdatwatcher: Save geographic info to allow for geographic analysis
3fc4cff analyse: Last changed datasets
83c04d5 Use the check message instead the plain URL
af32e09 Add a new analysis
09471df add a new helper function to delete Redis keys based on patterns
653849b enhance analyser
c59cba1 .gitignore: do not ignore heroku root go spec file
3db968d Merge branch 'master' of github.com:the42/ogdat
0982576 changed .gitignore
c83c4e4 enhance analyser functionality
69af533 drain the heartbeat channel so the heartbeat is actually written to the database
82ac6e5 go vet
4d1aa56 add database connection to analyser to read from Postgres
efacc85 embed database.DBConn so we can profit of derived methods
f143643 refactor Redis connection handle into common place
8355368 got fmt
6d404f2 add proper gitignore
893f274 add redis analyser package
03e2b5b refactor database functionality in own package
171b58f delete file
f4c262f never add .files to git
21bf3fa go vet
a55ca1d mention redis
67a4272 update schema to latest version as used in database
a09f3d3 no need for early exit on Heroku
fb3fa19 Procfile: s/ogdatwatcher/watcher
17a7c15 load timezone from environment
445a64f use an .env file to set the timezone relative to which the operations will be performend
e49d45c fix error in determining the last hittime from the database
70f9d3e ogdatwatcher
ded4d05 If there is nothing to do for more than 15 Minutes, exit
94d29e4 if the next action is more than <duration> away, exit
859021d If the server returns an error, it will not be parseable JSON, check for that
44ac9b6 check urls on sunday
539d8d6 urlchecktime: time is hard
dea987c select all fetchable urls, not those checked while last url check
245d7be use the first element to get the dataset-id.
e33915d only append a record if there are sub-records
a44d759 perform the url check on the coming day and coming hour.
18704cd no longer write a lockfile
b3ae10f temporary check on Heroku
f26c292 no need for a cast
b4e1471 ogdatwatcher:clamp data checks together by processing urls of a dataset together
1df0914 ogdatwatcher: better idle message
f73d986 ogdatwatcher:correctly set the timer for the datacheck
43372f3 improved scheduler logic
f5075e7 only check for new data after startup and once after every tick (not after each sleep periode)
2199135 check data and contact urls in periodic intervals and report result to database
4b6c5cf ogdatwatcher:factoring
0fdc639 select over channel, otherwise it will block
f58b0c0 beginning of checking of urls
371b293 s/CheckError/CheckInfo
424dab8 Merge branch 'master' of github.com:the42/ogdat
a60211f log to self: what to do about links
a16e98d ogdatjsonchecker: More information in messages
2e6030e a structural glitch is an info, not an error
e3c373e retain the inforamtion about a URL so we can periodically check
887a61e sql: schema changes
0fc7bc7 ogdatwatcher: save actual check status on field level as it contains more detailed information
42980b3 Check: discriminate on error type by using a bit set
b198541 Ugh, we check for required and optional fields, but when we write to the database we cannot be sure that all fields we intend to write are actually set
7b99912 more test cases
7af2f37 also support the case when category is directly specified as a string instead of an array
35bb7c3 Schedule: bugfix major flaw in process split logic
a6d65af cosmetic
3c6d44c fix: http.Get requires an absolute url
f1c4cd3 no longer support creating the database from the server process
6a67513 SQL schema and functions
2635ef8 get rid of some stored procedures
aab2499 improvements to ogdatwatcher
4475e1d use a time format CKAN actually understands
9f0a110 in order to use a map, it has to be created first
52ca39c some refactoring
6832ea0 go fmt
34e588b DBStringLen: Return string at max length x - subtle bug in here
966a112 get rid of a superfluous parameter
0325d2f major issue in calculating the run queue length fixed
f9f9de6 if a heartbeat fails, panic
8f75ad8 no need to trim description, it is a text field
e028848 simplify alive heartbeat logic
82d4221 fix typo
015d7b5 retry to get data just in case we are hitting a slow server or are on a slow connection
bb26795 error checking
ea04147 get rid of superfluous error handling
1c96030 bug haunting
d762103 haunting for bugs
6124183 follow links
0609ad4 Merge branch 'master' of heroku.com:guarded-tundra-1049
99bd737 check for error when creating a database transaction
8402c4f try error fix
35fdbaf add logic to process only metadata deltas
cd833c4 Watcher enhancements
4156dd4 get rid of one variable
37ce144 actually use the transaction
1228ab7 make consistent: receiver is of type pointer type
4f8bad1 go fmt
87ee7a5 refactoring
7c3a714 new package schedule
18760e2 refactoring and new functionality
481ad95 delete hidden and unused file
6265fda rename functions
586ced6 new symlinks
027bc89 enhancement to the watcher
af0e8dd Some encode Kategorie erroneously within a string. Be less anal about the JSON structure
0823dc2 further refactor and add database creation functionality
29200df rename file
067acb4 simplify and refactor code
f9505e4 simplify code
b1b5839 add test to check the urls as well
110271b add more unit tests
34e3257 add more unit tests
3bf2827 check maintainer_link also for sane link specifiers
4a71b59 improve checking for sane characters
30d05fe fix pacakge import
7f6d0af add more unit tests
ec6b9b6 minor code refactoring and better diagnostic message
6d234a3 Add information type (Warning, Error, ...) to diagnostic message
a385b9d more unit test
769c7d2 improve url checking
34dbc7c retain the information whether MetaData_Link was specified as a single element or as an array
e624a7d chew your own dogfood
8d79ee9 add helper function to check an array of strings against a case-insensitve, "-" stripped string
bfbdafa actually test empty array type
96cacb9 add more tests
8f25860 Treat json array types are empty when they are either null or contain no elements
780db0f minor unit test fixes
161509a assign static string outside of fmt.Sprintf function call
2109b01 more tests
1485540 CheckIANAEncoding now checks using IANA specified encodings in all lower case
ca48593 fetch IANA encodings, as they have been updated at 2013-01-23
8e73f47 add more tests
b3e3276 Improve tests for Check OGDATV21
94936bc fix slice index error in CheckOGDTextStringForSaneCharacters
86b6503 more test cases for required fields
805fd5b more test cases
84609c3 check email addresses case insensitive
1e81f9f * add number of resource which was checked to diagnostic messsage * fix a break to continue with label in checking for valid schema version * simplify code for checking for valid resource encoding
664bcc4 check emails case-insensitive
61d4ede use String regexp methods to avoid unnecessary string --> []byte conversions
7f878fd Check: actually compare results
8de1c79 ogdatjsonchecker: display number of errors found
ddb9ceb One field in testfile was missing -- added
895889b ignore Kate project files
dedb06c ogdatjson-checker improvements
bfbc1ff new test file
0c28d0b some fixes to Check:
4359768 reflect on required OGD fields and report if they are missing
4e58753 Merge branch 'master' of github.com:the42/ogdat
0100dcd check for nil elements in OGD struct members, even if they are mandatory
7f309b6 Merge branch 'master' of github.com:the42/ogdat
7d2ed0e fix number of arguments to print
7945a09 prettify error messages
a5a4bb5 report unicode replacement characters as invalid sequences.
23ae0a1 use 0 as offset instead of -1
b16a91d more tests
f697ce9 fix field check: inside the switch, a wrong field was accessed for inspection
485c987 pretty-print
f3d8d6c * add flag of a filename, into which the serialized OGD Metadata will be written * reorganize code, so that defered functions will fire and a status code is returned to the caller
4ffcaed move input to check (testfiles) into their own directory
2e8cea1 read tests as JSON-stream from file
625b67c Enhancements in understood command line switches
331ac6a Check returns an array of messages AND a generic error
c0f9542 use constants for type of check result Info, Warning, Error, StructuralError
e565448 it is sufficient to check for "http"
ba7dcaf go fmt
55ed907 add more information about errors in OGD field values
e0b5822 fix reflect mistakes and s/MetaDataStructFieldIsNil/IsNil
d1ae9f6 * rename MetaDataStructFieldIsNil --> IsNil * IsNil returns false, if value is not a pointer * GetBeschreibungForID: return struct and corresponding OGD_Kurzname
194ebbe symlink file
dff88af work around in JSON serialization error of data.gv.at
4cb6356 simplify metadata fetch logic
428085b add functionality
8bd2247 message unexpected OGD version of schema name
594c768 OGDAT V21 also checks OGDATV20
ed8589f completed Check()
bb02e49 also check encodings with "-" stripped
87a41f1 CheckIANAEncoding
ed96df7 IANA language file contains ambiguous entries, make them unique
3f682a0 make bash script to retrieve check utitlity files more robust
a71cab3 symlink for testing purpose
08a57cc rename IANA language file
d02dfad load iana encoding file
d3fe6c2 use function parameter to load file
fb5853b factor string message into constant
ea926ce reorder
4d3095a separate check and miscellaneous functionality of ogdat core packages into different files:
4d64455 mention requirements of shell script
f72d408 include all relevant files required to verify OGD data
8dc419c Check() - almost feature complete
fc1f27b move loading of ISO language file as an utility function into core package
7d9b010 remove ISO language file
3411a13 add more functionality to Check
5bed346 rename CheckLink to CheckUrlContact
5044478 initial code to check ressources
5cf51e3 complete checks in Check(). Still missing: Resources
f550c5b add a Raw field to cycle for unknown "Intervall" specifiers
177ca72 License_Citation is part of Extras, not Resource
f8d491e simplify code by getting rid of indention levels in if
dc34843 Check now takes a bool to indicate whether links should be tried to be retrieved
5f54522 add CheckLink function
7c2bfc6 symlinks should be relative
d9ee04a add a symlink
5dfe6ff move check functionality in core package
e43ae22 make a generic error type for Checks
b28c27c more symlinks for local executability
94fdd9f include copy of ISO language file
66e2448 extend Check
9bcc8bf clarify POLYGON WKT issue and adopt comment
3575b02 go vet
e59dbec s/MetaData/Metadata
9fc6676 add more Check functionality
5545531 Changes to time handling
01b5858 go vet discovered mistakes in struct tags
c92b0e5 refactor Check: generically check if a required field is not present
cdc3cab add new function returning if a value is nil. Will panic if value is not a pointer
25ed87a beginnings of Check for OGDAT V21
386272a make all struct members of OGDAT pointers
ea3b73c fix a typo s/Identfier/Identifier
17e6028 expand tests
f9363b2 strings.Index returns the index to the beginning of an occurrence of sub-string, have to add the length
4b46ba0 rename local variable from leftover functionality
2b5a9ac rename function
a794ad0 regenerate html spec of OGDAT V21
7ce4317 minor tweak to CSS in builtin template
a9426fa inital commit
e76057f beginnings of a MetaData Check routine
690485e no need to be overly verbose with the Version string
b351d89 minor cleanups
a1c2bfd rename function and concentrate functionality in package ogdat
00399e8 add struct tags to Metadata Sepcification to reflect a corresponding Beschreibung based on the ID
8763c5f add helper function which extracts an OGD-ID from a string
ee9be24 Add function GetOGDSetForVersion
25074eb Register now will return the OGD-AT Beschreibung.
ff17027 rename function and return OGDAT - Beschreibung and technical field name
7229556 Österreichische Open Government Metadatenkonvention als HTML
4cc685c minor tweaks in the built-in default html template
a9038a3 much simplify the builtin template
249df94 allow inclusion of htmlified version of the OGDAT Metadata specification
2a097f5 ignore html files (local tests)
aeda6c2 add an inline default template
90ca6dc default for boolean flag must be false, otherwise the programs default behavior would be to exit on invocation
7ba1b24 add labels of specification as a description of the set
23024a7 do not try to load an external template file as a default template but use pre-built internal. add a flag to print this internal template
42b3bb5 change the container type keeping an OGD-AT Beschreibung from a map to an ordered slice
ab1cb3c first open input file, then template, then output file
2cedbcf must not use the same name for a flag
49b9338 this is an executable, so name package must be "main"
ed0fdd1 initial commit of specification pretty printer
4909a97 as Loadspec is public, return an error. In case of error, the map will be nil
ecac40b export the functionality of loading a OGDAT CSV spec file
86b8817 go fix suggested the fix so do it
85607b3 symlink to ogdat V21 spec file for testing
72eb0cc move test for loading specification file into appropriate package
835c70b refactor code into shared package
8c45515 return specification for ID
a9ee9f0 go fmt
88fa09f What's inside and credit
641caff hand-tailored OGDAT V21 specification as CSV file
b093f5f auto generated files from pdftotext and awk script
b3910de shell script to automate downloading and extraction of OGDAT specification from PDF file
c38e432 awk script to extract the OGD-AT specifiation from a plain text representation of the PDF file
5f87612 git ignore pdf files
2e6f060 rename test function to what it is actually testing
9a8cca5 get rid of bufio reader to read first line, simply use an empty CSV read allow lazy quotes in strings. This is important as strings in the spec file are not delimited but contain intermediate quotes
b772864 only close the file once it got successfully opened
12da257 only close file once it was successfully opened skip reading first line as it contains the specification labels
e2b5e15 add ogdatv21 test file
9e128a5 actually try to load the specification file
4c825b3 move loading ISO language file into separate function for testing
6ee8c35 close ISO language file after reading
6c63f15 read OGD AT specification from file at init time
f1a9759 Beschreibung.Anzahl: change record type from int to byte Fix record specifier typo
c66d960 add column to distinguish required fields from optional ones
7d06abc fix minor typo - cosmentic
000d79d Split ogdat in description package and actual specification. New structure spupports versioning
2bddfb5 fix rename error of last rename
93bf4ac add a csv representation of OGD-AT Metadata specification
1161ee0 rename files, avoid underscores
5084e5e add version, cardinality and occurence(required, optional) to the OGD_AT field description
ed40210 add constants for language and encoding
2b14013 do not reuse local variable to set a global array. local will get re-used or recycled and thus cause errors in global variable
4685fdc parsing an url must not fail. if an error happens, save a nul pointer
a3dc90c complete OGD-AT matadata fields
62ae2b9 rename OGDTime to Time
c640154 change order of processing time related data
a20bfb1 move description into separate file
c26bbe8 log.Fatal will call os.Exit, what we not want as it prevents defered functions from running. Call log.Panic instead
3a8f9f0 new file: check OGD metadata consistency
46cdedd when dowloading the ISO language file a DB connection may not be required
bf7df6b add some more optional OGD fields
3b1d9a6 add Postgres database connectivity
6a76fcd reorder Katalog initialisation and prevent JSON serilisation of Katalog struct but ID value
39f0df4 ignore local foreman .env file
513404a command line parsing, lockfile handling, ISO language file download
5261405 ignore some local files created by the processes or for local development Procfile.local
ba27170 start the ogd watcher in serve mode
7ed10c6 silly typo in variable name
0c69c8e moving slowly
c90d4aa Procfile and .godir for Heroku
41321b6 Initial commit
REVERT: 5794d1c Update index.html
REVERT: 9efb0a7 added support for swagger-spec 1.2

git-subtree-dir: analyser/swagger-ui
git-subtree-split: cd2c75e74c3e3cf05f082bb64a3d872e5fffe542
Merge commit '9cd9860f9a370eaeb1b578b4fef1dbc1ef4f229a'
Update Swagger-UI
Squashed 'analyser/swagger-ui/' changes from cd2c75e..34bd474

34bd474 fix for #288
4601f62 fix for #334
2bddfb6 Merge pull request #327 from georgeOsdDev/master
b745801 Merge pull request #326 from aerotech/upstream
49f4804 Merge pull request #332 from thadudexx/master
e4d01c5 Update index.html
187517d fixed package to include less
846628f Match all image types
355380a Add image contents type resopnse handler
6db6300 fixing request_url updates for all operations in one resource
0505d6c Merge pull request #1 from wordnik/master
fcb82bd merged https://github.com/wordnik/swagger-ui/pull/323
348f1e1 updated swagger-js
abc3600 Merge pull request #295 from tomrac87/hal_json
77882a0 Merge pull request #311 from aerotech/master
0e98639 updating docs, change in sample and java required for build
3347d7e merged pr for https://github.com/wordnik/swagger-js/pull/54
929f8b6 Merge branch 'master' of github.com:wordnik/swagger-ui
8158e9b fix for #310
a0db858 fixed example
1b662db fix for required fields showing optional
cd4fa99 added resource description
bc93240 fixed merge issue
9993675 fix for #301, headers being URI encoded
8a9eedf manual merge of #304
046e534 manual merge of #303
73c89ba fix for #297, enabled throbber
ff05f91 Added proper formatting for hal+json Content-Type responses in coffeescript source file
6c1de0b added enum support per #296
4fb5b86 updated swagger-js to address sample schema issues
5f4eec1 Added proper formatting for hal+json Content-Type responses
f22efb6 fix for #279, empty boolean drop-downs
9e25ec5 Merge branch 'master' of github.com:wordnik/swagger-ui
085cc05 fixed responseMessages issue per #267
56a5814 Merge pull request #287 from rcrooks/master
7802c36 fixing file name -- realized the typo in the link tag was replicating one in the css file name
fca0bdf Merge pull request #286 from bcls/master
727d9e0 fix for issue #285
888db09 fixed enum, required flags
f29d53d updated readme
b138359 updated readme
9fccf1c updated swagger-client
25805a4 fixed petstore link
6865ab5 merged from develop branch
085fc8d updated handlebars
205125c fixes for content-type
4b5e4e5 updated for 1.2 support
b2637fc removed console log, updated swagger.js
3c3e297 Merge pull request #272 from thinkt4nk/master
dd88dc1 fixed discrepancy between handlebars precompiler and runtime
c6f2aad merged https://github.com/wordnik/swagger-js/issues/42
df20d2b fix for #263
e72dbd6 fix for #261
f4d77d7 fix for #261
390ee1c removed console logs
627fd8b removed some logging
078851b fixed default URL
e980cca added api info
187d245 updated to use petstore instead of localhost
5874615 added authorization support from swagger-js 2.0
2a206b0 renamed discoveryUrl to url to match js change
6a5dfdc fix for #174, #78
8f7e8f9 updated to latest swagger-js
22e5f5c #199
8fcaa60 removed swagger-client dep for now, it's copied manually
8bac45d updated deps, version
09be20c merged with swagger.js-2.0-develop
7334079 added shred library
cde29f0 added separate request and response templates
7b9f8a2 added separate request and response templates
2dd2eb9 added shred library
4a6fc1d added shred library
5794d1c Update index.html
9efb0a7 added support for swagger-spec 1.2
REVERT: cd2c75e expected 6 SQL fields, but got only 5 - fix SQL statement
REVERT: ec209a0 add full url to the check message to get more helpful debugging text
REVERT: 43e252d AN001 was erroneous by SQL, fix it
REVERT: 4a1a142 SQL query optimization. Now takes 1/4 of the time
REVERT: 2feea30 deleted datasets need not get checked
REVERT: c400e1d new API function
REVERT: b4b2217 fix typo in when rendering the OGDAT-metadata specification
REVERT: 087fde1 add a new route
REVERT: 2c906ae Merge branch 'master' of github.com:the42/ogdat
REVERT: 41f07f1 Fix NOT NULL definition
REVERT: a5d60be common place for redis keys
REVERT: a87d0a2 new functionality: Add DateSets based on their taxonomy membership
REVERT: 2075511 redis.AddFlat is meaningless as it uses fmt.Print(#v)
REVERT: e7a43f8 new version of go-restful handles flat / gzip encoding in the Container
REVERT: e54864e Keep serving the API even the analysing functionality is disabled
REVERT: ec3a4e3 slightly rebuild the logic how taxonomies are stored in redis and retrieved via the API
REVERT: d3e72db add documentation about the currently naive approach to rebuild statistics upon data change
REVERT: 1258fbd Execute the http server in its own go-routine
REVERT: db43409 remove debugging code
REVERT: 6cbe221 enable XHR CORS OPTIONS
REVERT: 9183ff0 test why a channel doesn't get called
REVERT: a3f39dd an API base path may be empty
REVERT: 6ca1f9b Get API base path from environemnt variable
REVERT: aa4d0b5 for configuration, no local port required
REVERT: a26f38a rename environment variable
REVERT: 3cb7834 changes due to update of go-restful
REVERT: 48f3d0a read hostname from environemnt variable
REVERT: 7b5148d prepare for public availability
REVERT: 6cefbd9 add check results to redis database
REVERT: b235cb1 there is no need for this code any longer since the UI lives in its own repository
REVERT: 2af8b33 delete user interface component. It is now available as github.com/the42/ogdat-analyserui
REVERT: e5e40fc Merge branch 'master' of github.com:the42/ogdat
REVERT: ebc47f1 Merge branch 'master' of github.com:the42/ogdat
REVERT: 5135448 DRY - use an anoynmous function to prevent repetition in a select case
REVERT: 756bd41 commit 2/2: no versioning in API urls
REVERT: 0dd0ffd commit 1/2: no versioning in API urls
REVERT: 5be5a38 Service is available with an api version specifier
REVERT: 3ff337b Merge commit '6c0dd8b39114c746c4e4c0280a554787922b91aa'
REVERT: 6c0dd8b Squashed 'analyser/swagger-ui/' changes from dd3037f..5794d1c
REVERT: 4d066ce Beginnings of the actual analyse-UI
REVERT: 9106d37 new url for dynamic documentation
REVERT: 92e2c56 prepare for heroku
REVERT: dbaf019 CORS-enable the API
REVERT: 03f9c05 instead of panicking on a fatal error, set the appropriate response code
REVERT: 2a3edfb gzip-encode the response
REVERT: e0eadea in case of error, quit the listen event loop
REVERT: 6a5b0d8 guarantee that redis keys are truly unique
REVERT: d037a8e make retrieval of sorted sets generic
REVERT: 0be4006 make retrieval of sorted sets generic
REVERT: 3442e30 convert redis connection handling to using a pool
REVERT: 1115d54 changes, enhancements and refactoring
REVERT: 64281c9 rename file
REVERT: f6fa6db In our case, git submodules are not a good idea, use subtrees instead.
REVERT: 16b3ae6 Add 'analyser/swagger-ui/' from commit 'dd3037f481979ec15b8f1c19154d800565622a53'
REVERT: dca1d7c submodules are not the way to go
REVERT: 913d6b6 no need for a closure, cf. https://github.com/emicklei/go-restful/issues/21#issuecomment-20252820
REVERT: 2170f48 First stab at the restful API for Angular frontend
REVERT: 2d47536 add swagger-ui submodule
REVERT: 67d711d Move code arround. No new functionality or additions.
REVERT: 97b1228 Publish a data change event into Redis
REVERT: 2266729 add Redis Publish helper function
REVERT: 6399b39 in case of error, return error, not nil
REVERT: 6f9eff9 analyser: Populate geographic information into Redis
REVERT: 2bafb0e change sql schema to reflect changes
REVERT: 86a114e ogdatwatcher: Save geographic info to allow for geographic analysis
REVERT: 3fc4cff analyse: Last changed datasets
REVERT: 83c04d5 Use the check message instead the plain URL
REVERT: af32e09 Add a new analysis
REVERT: 09471df add a new helper function to delete Redis keys based on patterns
REVERT: 653849b enhance analyser
REVERT: c59cba1 .gitignore: do not ignore heroku root go spec file
REVERT: 3db968d Merge branch 'master' of github.com:the42/ogdat
REVERT: 0982576 changed .gitignore
REVERT: c83c4e4 enhance analyser functionality
REVERT: 69af533 drain the heartbeat channel so the heartbeat is actually written to the database
REVERT: 82ac6e5 go vet
REVERT: 4d1aa56 add database connection to analyser to read from Postgres
REVERT: efacc85 embed database.DBConn so we can profit of derived methods
REVERT: f143643 refactor Redis connection handle into common place
REVERT: 8355368 got fmt
REVERT: 6d404f2 add proper gitignore
REVERT: 893f274 add redis analyser package
REVERT: 03e2b5b refactor database functionality in own package
REVERT: 171b58f delete file
REVERT: f4c262f never add .files to git
REVERT: 21bf3fa go vet
REVERT: a55ca1d mention redis
REVERT: 67a4272 update schema to latest version as used in database
REVERT: a09f3d3 no need for early exit on Heroku
REVERT: fb3fa19 Procfile: s/ogdatwatcher/watcher
REVERT: 17a7c15 load timezone from environment
REVERT: 445a64f use an .env file to set the timezone relative to which the operations will be performend
REVERT: e49d45c fix error in determining the last hittime from the database
REVERT: 70f9d3e ogdatwatcher
REVERT: ded4d05 If there is nothing to do for more than 15 Minutes, exit
REVERT: 94d29e4 if the next action is more than <duration> away, exit
REVERT: 859021d If the server returns an error, it will not be parseable JSON, check for that
REVERT: 44ac9b6 check urls on sunday
REVERT: 539d8d6 urlchecktime: time is hard
REVERT: dea987c select all fetchable urls, not those checked while last url check
REVERT: 245d7be use the first element to get the dataset-id.
REVERT: e33915d only append a record if there are sub-records
REVERT: a44d759 perform the url check on the coming day and coming hour.
REVERT: 18704cd no longer write a lockfile
REVERT: b3ae10f temporary check on Heroku
REVERT: f26c292 no need for a cast
REVERT: b4e1471 ogdatwatcher:clamp data checks together by processing urls of a dataset together
REVERT: 1df0914 ogdatwatcher: better idle message
REVERT: f73d986 ogdatwatcher:correctly set the timer for the datacheck
REVERT: 43372f3 improved scheduler logic
REVERT: f5075e7 only check for new data after startup and once after every tick (not after each sleep periode)
REVERT: 2199135 check data and contact urls in periodic intervals and report result to database
REVERT: 4b6c5cf ogdatwatcher:factoring
REVERT: 0fdc639 select over channel, otherwise it will block
REVERT: f58b0c0 beginning of checking of urls
REVERT: 371b293 s/CheckError/CheckInfo
REVERT: 424dab8 Merge branch 'master' of github.com:the42/ogdat
REVERT: a60211f log to self: what to do about links
REVERT: a16e98d ogdatjsonchecker: More information in messages
REVERT: 2e6030e a structural glitch is an info, not an error
REVERT: e3c373e retain the inforamtion about a URL so we can periodically check
REVERT: 887a61e sql: schema changes
REVERT: 0fc7bc7 ogdatwatcher: save actual check status on field level as it contains more detailed information
REVERT: 42980b3 Check: discriminate on error type by using a bit set
REVERT: b198541 Ugh, we check for required and optional fields, but when we write to the database we cannot be sure that all fields we intend to write are actually set
REVERT: 7b99912 more test cases
REVERT: 7af2f37 also support the case when category is directly specified as a string instead of an array
REVERT: 35bb7c3 Schedule: bugfix major flaw in process split logic
REVERT: a6d65af cosmetic
REVERT: 3c6d44c fix: http.Get requires an absolute url
REVERT: f1c4cd3 no longer support creating the database from the server process
REVERT: 6a67513 SQL schema and functions
REVERT: 2635ef8 get rid of some stored procedures
REVERT: aab2499 improvements to ogdatwatcher
REVERT: 4475e1d use a time format CKAN actually understands
REVERT: 9f0a110 in order to use a map, it has to be created first
REVERT: 52ca39c some refactoring
REVERT: 6832ea0 go fmt
REVERT: 34e588b DBStringLen: Return string at max length x - subtle bug in here
REVERT: 966a112 get rid of a superfluous parameter
REVERT: 0325d2f major issue in calculating the run queue length fixed
REVERT: f9f9de6 if a heartbeat fails, panic
REVERT: 8f75ad8 no need to trim description, it is a text field
REVERT: e028848 simplify alive heartbeat logic
REVERT: 82d4221 fix typo
REVERT: 015d7b5 retry to get data just in case we are hitting a slow server or are on a slow connection
REVERT: bb26795 error checking
REVERT: ea04147 get rid of superfluous error handling
REVERT: 1c96030 bug haunting
REVERT: d762103 haunting for bugs
REVERT: 6124183 follow links
REVERT: 0609ad4 Merge branch 'master' of heroku.com:guarded-tundra-1049
REVERT: 99bd737 check for error when creating a database transaction
REVERT: 8402c4f try error fix
REVERT: 35fdbaf add logic to process only metadata deltas
REVERT: cd833c4 Watcher enhancements
REVERT: 4156dd4 get rid of one variable
REVERT: 37ce144 actually use the transaction
REVERT: 1228ab7 make consistent: receiver is of type pointer type
REVERT: 4f8bad1 go fmt
REVERT: 87ee7a5 refactoring
REVERT: 7c3a714 new package schedule
REVERT: 18760e2 refactoring and new functionality
REVERT: 481ad95 delete hidden and unused file
REVERT: 6265fda rename functions
REVERT: 586ced6 new symlinks
REVERT: 027bc89 enhancement to the watcher
REVERT: af0e8dd Some encode Kategorie erroneously within a string. Be less anal about the JSON structure
REVERT: 0823dc2 further refactor and add database creation functionality
REVERT: 29200df rename file
REVERT: 067acb4 simplify and refactor code
REVERT: f9505e4 simplify code
REVERT: b1b5839 add test to check the urls as well
REVERT: 110271b add more unit tests
REVERT: 34e3257 add more unit tests
REVERT: 3bf2827 check maintainer_link also for sane link specifiers
REVERT: 4a71b59 improve checking for sane characters
REVERT: 30d05fe fix pacakge import
REVERT: 7f6d0af add more unit tests
REVERT: ec6b9b6 minor code refactoring and better diagnostic message
REVERT: 6d234a3 Add information type (Warning, Error, ...) to diagnostic message
REVERT: a385b9d more unit test
REVERT: 769c7d2 improve url checking
REVERT: 34dbc7c retain the information whether MetaData_Link was specified as a single element or as an array
REVERT: e624a7d chew your own dogfood
REVERT: 8d79ee9 add helper function to check an array of strings against a case-insensitve, "-" stripped string
REVERT: bfbdafa actually test empty array type
REVERT: 96cacb9 add more tests
REVERT: 8f25860 Treat json array types are empty when they are either null or contain no elements
REVERT: 780db0f minor unit test fixes
REVERT: 161509a assign static string outside of fmt.Sprintf function call
REVERT: 2109b01 more tests
REVERT: 1485540 CheckIANAEncoding now checks using IANA specified encodings in all lower case
REVERT: ca48593 fetch IANA encodings, as they have been updated at 2013-01-23
REVERT: 8e73f47 add more tests
REVERT: b3e3276 Improve tests for Check OGDATV21
REVERT: 94936bc fix slice index error in CheckOGDTextStringForSaneCharacters
REVERT: 86b6503 more test cases for required fields
REVERT: 805fd5b more test cases
REVERT: 84609c3 check email addresses case insensitive
REVERT: 1e81f9f * add number of resource which was checked to diagnostic messsage * fix a break to continue with label in checking for valid schema version * simplify code for checking for valid resource encoding
REVERT: 664bcc4 check emails case-insensitive
REVERT: 61d4ede use String regexp methods to avoid unnecessary string --> []byte conversions
REVERT: 7f878fd Check: actually compare results
REVERT: 8de1c79 ogdatjsonchecker: display number of errors found
REVERT: ddb9ceb One field in testfile was missing -- added
REVERT: 895889b ignore Kate project files
REVERT: dedb06c ogdatjson-checker improvements
REVERT: bfbc1ff new test file
REVERT: 0c28d0b some fixes to Check:
REVERT: 4359768 reflect on required OGD fields and report if they are missing
REVERT: 4e58753 Merge branch 'master' of github.com:the42/ogdat
REVERT: 0100dcd check for nil elements in OGD struct members, even if they are mandatory
REVERT: 7f309b6 Merge branch 'master' of github.com:the42/ogdat
REVERT: 7d2ed0e fix number of arguments to print
REVERT: 7945a09 prettify error messages
REVERT: a5a4bb5 report unicode replacement characters as invalid sequences.
REVERT: 23ae0a1 use 0 as offset instead of -1
REVERT: b16a91d more tests
REVERT: f697ce9 fix field check: inside the switch, a wrong field was accessed for inspection
REVERT: 485c987 pretty-print
REVERT: f3d8d6c * add flag of a filename, into which the serialized OGD Metadata will be written * reorganize code, so that defered functions will fire and a status code is returned to the caller
REVERT: 4ffcaed move input to check (testfiles) into their own directory
REVERT: 2e8cea1 read tests as JSON-stream from file
REVERT: 625b67c Enhancements in understood command line switches
REVERT: 331ac6a Check returns an array of messages AND a generic error
REVERT: c0f9542 use constants for type of check result Info, Warning, Error, StructuralError
REVERT: e565448 it is sufficient to check for "http"
REVERT: ba7dcaf go fmt
REVERT: 55ed907 add more information about errors in OGD field values
REVERT: e0b5822 fix reflect mistakes and s/MetaDataStructFieldIsNil/IsNil
REVERT: d1ae9f6 * rename MetaDataStructFieldIsNil --> IsNil * IsNil returns false, if value is not a pointer * GetBeschreibungForID: return struct and corresponding OGD_Kurzname
REVERT: 194ebbe symlink file
REVERT: dff88af work around in JSON serialization error of data.gv.at
REVERT: 4cb6356 simplify metadata fetch logic
REVERT: 428085b add functionality
REVERT: 8bd2247 message unexpected OGD version of schema name
REVERT: 594c768 OGDAT V21 also checks OGDATV20
REVERT: ed8589f completed Check()
REVERT: bb02e49 also check encodings with "-" stripped
REVERT: 87a41f1 CheckIANAEncoding
REVERT: ed96df7 IANA language file contains ambiguous entries, make them unique
REVERT: 3f682a0 make bash script to retrieve check utitlity files more robust
REVERT: a71cab3 symlink for testing purpose
REVERT: 08a57cc rename IANA language file
REVERT: d02dfad load iana encoding file
REVERT: d3fe6c2 use function parameter to load file
REVERT: fb5853b factor string message into constant
REVERT: ea926ce reorder
REVERT: 4d3095a separate check and miscellaneous functionality of ogdat core packages into different files:
REVERT: 4d64455 mention requirements of shell script
REVERT: f72d408 include all relevant files required to verify OGD data
REVERT: 8dc419c Check() - almost feature complete
REVERT: fc1f27b move loading of ISO language file as an utility function into core package
REVERT: 7d9b010 remove ISO language file
REVERT: 3411a13 add more functionality to Check
REVERT: 5bed346 rename CheckLink to CheckUrlContact
REVERT: 5044478 initial code to check ressources
REVERT: 5cf51e3 complete checks in Check(). Still missing: Resources
REVERT: f550c5b add a Raw field to cycle for unknown "Intervall" specifiers
REVERT: 177ca72 License_Citation is part of Extras, not Resource
REVERT: f8d491e simplify code by getting rid of indention levels in if
REVERT: dc34843 Check now takes a bool to indicate whether links should be tried to be retrieved
REVERT: 5f54522 add CheckLink function
REVERT: 7c2bfc6 symlinks should be relative
REVERT: d9ee04a add a symlink
REVERT: 5dfe6ff move check functionality in core package
REVERT: e43ae22 make a generic error type for Checks
REVERT: b28c27c more symlinks for local executability
REVERT: 94fdd9f include copy of ISO language file
REVERT: 66e2448 extend Check
REVERT: 9bcc8bf clarify POLYGON WKT issue and adopt comment
REVERT: 3575b02 go vet
REVERT: e59dbec s/MetaData/Metadata
REVERT: 9fc6676 add more Check functionality
REVERT: 5545531 Changes to time handling
REVERT: 01b5858 go vet discovered mistakes in struct tags
REVERT: c92b0e5 refactor Check: generically check if a required field is not present
REVERT: cdc3cab add new function returning if a value is nil. Will panic if value is not a pointer
REVERT: 25ed87a beginnings of Check for OGDAT V21
REVERT: 386272a make all struct members of OGDAT pointers
REVERT: ea3b73c fix a typo s/Identfier/Identifier
REVERT: 17e6028 expand tests
REVERT: f9363b2 strings.Index returns the index to the beginning of an occurrence of sub-string, have to add the length
REVERT: 4b46ba0 rename local variable from leftover functionality
REVERT: 2b5a9ac rename function
REVERT: a794ad0 regenerate html spec of OGDAT V21
REVERT: 7ce4317 minor tweak to CSS in builtin template
REVERT: a9426fa inital commit
REVERT: e76057f beginnings of a MetaData Check routine
REVERT: 690485e no need to be overly verbose with the Version string
REVERT: b351d89 minor cleanups
REVERT: a1c2bfd rename function and concentrate functionality in package ogdat
REVERT: 00399e8 add struct tags to Metadata Sepcification to reflect a corresponding Beschreibung based on the ID
REVERT: 8763c5f add helper function which extracts an OGD-ID from a string
REVERT: ee9be24 Add function GetOGDSetForVersion
REVERT: 25074eb Register now will return the OGD-AT Beschreibung.
REVERT: ff17027 rename function and return OGDAT - Beschreibung and technical field name
REVERT: 7229556 Österreichische Open Government Metadatenkonvention als HTML
REVERT: 4cc685c minor tweaks in the built-in default html template
REVERT: a9038a3 much simplify the builtin template
REVERT: 249df94 allow inclusion of htmlified version of the OGDAT Metadata specification
REVERT: 2a097f5 ignore html files (local tests)
REVERT: aeda6c2 add an inline default template
REVERT: 90ca6dc default for boolean flag must be false, otherwise the programs default behavior would be to exit on invocation
REVERT: 7ba1b24 add labels of specification as a description of the set
REVERT: 23024a7 do not try to load an external template file as a default template but use pre-built internal. add a flag to print this internal template
REVERT: 42b3bb5 change the container type keeping an OGD-AT Beschreibung from a map to an ordered slice
REVERT: ab1cb3c first open input file, then template, then output file
REVERT: 2cedbcf must not use the same name for a flag
REVERT: 49b9338 this is an executable, so name package must be "main"
REVERT: ed0fdd1 initial commit of specification pretty printer
REVERT: 4909a97 as Loadspec is public, return an error. In case of error, the map will be nil
REVERT: ecac40b export the functionality of loading a OGDAT CSV spec file
REVERT: 86b8817 go fix suggested the fix so do it
REVERT: 85607b3 symlink to ogdat V21 spec file for testing
REVERT: 72eb0cc move test for loading specification file into appropriate package
REVERT: 835c70b refactor code into shared package
REVERT: 8c45515 return specification for ID
REVERT: a9ee9f0 go fmt
REVERT: 88fa09f What's inside and credit
REVERT: 641caff hand-tailored OGDAT V21 specification as CSV file
REVERT: b093f5f auto generated files from pdftotext and awk script
REVERT: b3910de shell script to automate downloading and extraction of OGDAT specification from PDF file
REVERT: c38e432 awk script to extract the OGD-AT specifiation from a plain text representation of the PDF file
REVERT: 5f87612 git ignore pdf files
REVERT: 2e6f060 rename test function to what it is actually testing
REVERT: 9a8cca5 get rid of bufio reader to read first line, simply use an empty CSV read allow lazy quotes in strings. This is important as strings in the spec file are not delimited but contain intermediate quotes
REVERT: b772864 only close the file once it got successfully opened
REVERT: 12da257 only close file once it was successfully opened skip reading first line as it contains the specification labels
REVERT: e2b5e15 add ogdatv21 test file
REVERT: 9e128a5 actually try to load the specification file
REVERT: 4c825b3 move loading ISO language file into separate function for testing
REVERT: 6ee8c35 close ISO language file after reading
REVERT: 6c63f15 read OGD AT specification from file at init time
REVERT: f1a9759 Beschreibung.Anzahl: change record type from int to byte Fix record specifier typo
REVERT: c66d960 add column to distinguish required fields from optional ones
REVERT: 7d06abc fix minor typo - cosmentic
REVERT: 000d79d Split ogdat in description package and actual specification. New structure spupports versioning
REVERT: 2bddfb5 fix rename error of last rename
REVERT: 93bf4ac add a csv representation of OGD-AT Metadata specification
REVERT: 1161ee0 rename files, avoid underscores
REVERT: 5084e5e add version, cardinality and occurence(required, optional) to the OGD_AT field description
REVERT: ed40210 add constants for language and encoding
REVERT: 2b14013 do not reuse local variable to set a global array. local will get re-used or recycled and thus cause errors in global variable
REVERT: 4685fdc parsing an url must not fail. if an error happens, save a nul pointer
REVERT: a3dc90c complete OGD-AT matadata fields
REVERT: 62ae2b9 rename OGDTime to Time
REVERT: c640154 change order of processing time related data
REVERT: a20bfb1 move description into separate file
REVERT: c26bbe8 log.Fatal will call os.Exit, what we not want as it prevents defered functions from running. Call log.Panic instead
REVERT: 3a8f9f0 new file: check OGD metadata consistency
REVERT: 46cdedd when dowloading the ISO language file a DB connection may not be required
REVERT: bf7df6b add some more optional OGD fields
REVERT: 3b1d9a6 add Postgres database connectivity
REVERT: 6a76fcd reorder Katalog initialisation and prevent JSON serilisation of Katalog struct but ID value
REVERT: 39f0df4 ignore local foreman .env file
REVERT: 513404a command line parsing, lockfile handling, ISO language file download
REVERT: 5261405 ignore some local files created by the processes or for local development Procfile.local
REVERT: ba27170 start the ogd watcher in serve mode
REVERT: 7ed10c6 silly typo in variable name
REVERT: 0c69c8e moving slowly
REVERT: c90d4aa Procfile and .godir for Heroku
REVERT: 41321b6 Initial commit

git-subtree-dir: analyser/swagger-ui
git-subtree-split: 34bd474bffe1883c553990f792df63af37f18af2
Update for latest swagger-ui

Merge commit '2b08801285b08ea0ef97a7c0973d9f922d8df842'
use our default url for swagger