Package: meine-stadt-transparent
Meine Stadt TransparentMeine Stadt Transparent is a free council information system. Its current main focus is presenting data from offical German council information systems, so called "Ratsinformationssysteme". Those are imported using theOParlAPI, which can easily customized. You can even write your own importer for arbitrary data sources.Our sample live system using the data of the cityKrefeldis available at:https://krefeld.meine-stadt-transparent.de/. We provide a public chat on riot at#meine-stadt-transparent:matrix.org, which you can join onmatrix.The project was sponsored by thePrototype Fund.About this projectMeine Stadt Transparent makes decision-making in city councils and administrations more transparent by providing easy access to information about the city council, including published documents, motions and meeting agendas. As a successor to Munich'sMünchen Transparent, its aim is to be easily deployable for as many cities as possible.It includes many features regarding data research and staying up to date, targeted both towards citizens and journalists:Information about city councillors, administrative organizations and meetings of the city council are provided.All published documents are searchable in a flexible manner, be it motions, resolutions, meeting agendas or protocols. The search supports both simple full-text searches and flexible criteria-based filters.Documents are automatically searched for mentioned places. A map is provided indicating places that are mentioned. Thus, it is easy to identify documents that affect places in your living neighborhood.You can subscribe to topics / search expressions to get notified by e-mail, once new documents matching your query are published.It supports several ways of subscribing to new content: different kinds of RSS-feeds and subscribing to the meeting calendar using the iCal-format.We try to make Meine Stadt Transparent accessible by everyone: the layout is responsive to provide a good experience on mobile device, and we follow accessibility standards (WCAG 2.0 AA, ARIA) as close as possible.Meine Stadt Transparent isnota complete replacement for traditional council information systems, however: it focuses on displaying already published information to the public. It does not provide a user-accessible backend for content authoring. It relies on the availability of an API provided by a council information system backend. Currently, the openOparl-Standardis supported.Production setup with docker composePrerequisites: A host with root access and enough ram for elasticsearch and mariadb. If you don't have much ram, create a big swapfile for memory spikes in the import.All services will run in docker containers orchestrated by docker compose, with nginx as reverse proxy in front of them which also serves static files.First, installdockeranddocker compose. Thenadjust max_map_counton the host system for elasticsearch.Downloadetc/docker-compose.ymlfrom the root of this repository. Replace allchangemewith real random passwords (Hint:openssl rand -hex 32).Downloadetc/template.envto.env. ChangeREAL_HOSTto your domain,SECRET_KEYto a randomly generated secret and use the same passwords as indocker-compose.ymlforDATABASE_URL, andMINIO_SECRET_KEY. You most likely want to configure third-party services as described later, but you can postpone that until after the base site works.To deliver the assets through nginx, we need to mount them to a local container:mkdir log
chown 33:33 log
rm -rf /var/www/meine-stadt-transparent-static # Delete existing or it will land in a subdirectory 
docker cp django:/static /var/www/meine-stadt-transparent-staticYou can change the directory to any other as long as you match that later in your nginx conf.Start everything:docker-compose upWait until the elasticsearch log saysCluster health status changed from [RED] to [YELLOW]and open another terminal. You can later start the services as daemons with-dor stop them withdocker-compose down.Then we can run the migrations, create the buckets for minio (our file storage) and create the elasticsearch indices. If something failed you can rerun the setup command, it will only create missing indices.docker-compose run --rm django ./manage.py setupLet's load some dummy data to check everythings wokring:docker-compose run --rm django ./manage.py loaddata mainapp/fixtures/initdata.jsonYou should now get a 200 response fromlocalhost:8000.If you've not familiar with nginx, you should start withthis tutorial. Install nginx,certbotand the certbot nginx integration. For ubuntu it is e.g.sudo add-apt-repository ppa:certbot/certbot
sudo apt update
sudo apt install nginx certbot python3-certbot-nginxDownloadetc/nginx-http.conf, add it to your nginx sites and replacechangeme.tldwith your domain. Then run certbot and follow the instructions:certbot --nginxCertbot will rewrite the nginx configuration to a version with strong encryption. You might also want to activate http/2 by addinghttp2after both443 ssl.You now have a proper site at your domain!Now that everything is in place, drop the dummy data:docker-compose run --rm django ./manage.py flushInstead, import real data by replacingSpringfieldwith the name of your city. Seedocs/Import.mdfor details.docker-compose run --rm django ./manage.py import SpringfieldYou should now have a usable instance!Finally, create a daily cronjob with the following. This will import changed objects from the oparl api and then notify the users. Also make sure that there is a cronjob for certbot.docker-compose run --rm django ./manage.py cronYou can execute all the other commands from this readme by prepending them withdocker-compose run --rm django(or starting a shell in the container). Note for advanced users:.venv/bin/pythonis configured as entrypoint.Next, have a look atdocs/Customization.md.UpdatesAfter pulling a new version of the docker container, you need to run the following commands to update the assets:docker-compose down
rm -r /var/www/meine-stadt-transparent-static
mkdir /var/www/meine-stadt-transparent-static
docker-compose run --rm django ./manage.py setup
docker-compose up -dKubernetesIf you have a Kubernetes cluster, you can have a look atthis experimental setupwhich is used by Münster.Manual SetupRequirementsPython 3.8, 3.9 or 3.10 with pip andpoetry1.1A recent node version (v16) with npm (v8)A webserver (nginx or apache is recommended)A Database (MariaDB is recommended, though anything that django supports should work)minioIf you want to use elasticsearch, you either needdocker and docker composeor will have toinstall elasticsearch 7.9 yourselfOn Debian/Ubuntu:sudo apt install python3-pip python3-venv python3-dev nodejs \
    git libmysqlclient-dev libmagickwand-dev poppler-utils libssl-dev gettextInstall dependencies.poetry config virtualenvs.in-project true # This is not mandatory, yet quite useful
poetry install
npm installActivate the virtualenv created by poetry. You either need to run this in your shell before running any other python command or prefix any python command withpoetry run.poetry shellCopyetc/template.envto.envand adjust the values. You can specify a different dotenv file with theENV_PATHenvironment variable.Configure your webserver, see e.g.etc/nginx.confProductionThe following steps are only required when you want to deploy the site to production. For development, see the corresponding section belownpm run build:prod
npm run build:email
./manage.py collectstaticFollow thethe official guide. Unlike the guide, we recommend gunicorn over wsgi as gunicorn is much simpler to configure.The site is now ready :tada:. Next, have a look atdocs/Customization.mdanddocs/Import.md.DevelopmentPlease refer todocs/Development.mdKnown ProblemsIf you hit problems regarding memory when starting elasticsearch, please have a look at thisdocumentation.If MySQL/MariaDB is to be used as a database backend, a Version of at least 5.7 (MySQL) or 10.2 (MariaDB) is needed, with Barracuda being set as the default format for new InnoDB-Tables (default), otherwise you will run into errors about too long Indexes.LicenseThis software is published under the terms of the MIT license. The json files undertestdata/oparlare adapted from the oparl project and licensed under CC-BY-SA-4.0. The license of the included animal picturesmainapp/assets/imagesare CC0 and CC-BY-SA Luigi Rosa. The redistribution ofetc/Donald Knuth - The Complexity of Songs.pdfis explicitly allowed in its last paragraph.