As Senior Software Engineer at company building messaging platform for healthcare industry I am responsible, including other duties, for performance of our application. We develop pretty standard web-service using Ruby on Rails application for business logic and API, React + Redux for users' facing single page application, as database we use PostgreSQL. Common reasons for performance problems in similar stacks are heavy queries to database and I would like to tell the story how we applied non-standard but fairly simple optimisations to improve performance. Our business operates in US, so we have to be HIPAA compliant and follow certain security policies, security audit is something that we are always prepared for. To reduce risks and costs we rely on a special cloud provider to run our applications and databases, very similar to what Heroku does. On one hand it allows us to focus on building our platform but on the other hand it adds an additional limitation to our infrastructure. Talking shortly — we cannot scale up infinitely. As a successful startup we double number of users every few month and one day our monitoring told us that we were exceeding disk IO quota on the database server. Underlying AWS started throttling which was resulting in a significant performance degradation. Ruby application was not capable to serve all incoming traffic because Unicorn workers were spending too much time awaiting for database's response, customers were unhappy. In the beginning of the article I mentioned the phrase "non standard optimisations" because all low-hanging fruits were already picked: All these tricks did a great job for a couple of month until we faced again the same performance problem — more users, higher load. We were looking for something else. We did not want to use heavy artillery and implement denormalisation and partitioning because these solutions require deep knowledge in databases, shift team's focus from implementing features to maintenance and by the end we wanted to avoid complexity in our application. Lastly, we used PostgreSQL 9.3 where partitions are based on triggers with all their costs. KISS principle in action. We decided to focus on the main symptom — disk IO. As less data we store, as less IO capacity we need, this was the primary idea. We started looking for opportunities to compress data and the first candidates were columns like user_type provided with polymorphic associations by ActiveRecord. In the application we use modules a lot which lead us to have long strings like Module::SubModule::ModelName for polymorphic associations. What we did — convert all these columns' types from varchar to ENUM. The Rails migration looks like this: A few notes about this migration for people who not familiar with Rails: Needless to say that we run all similar migrations during lowest activity periods with proper testing. We converted all polymorphic columns into ENUM, dropped old columns after a few days of monitoring and finally run VACUUM to decrease fragmentation. This saved us roughly 10% of total disk space but
some tables with a few columns were compressed twice! What was more important — some tables started to be cached in memory (remember, we cannot easily add more RAM) by PostgreSQL and this dramatically decreased the required disk IO. Another thing was found in the article How a single PostgreSQL config change improved slow query performance by 50x — our PostgreSQL provider makes an automatic configuration for server based on requested volume of RAM, Disk and CPU but by whatever reason they left the parameter random_page_cost with the default value which is 4 optimised for HDD. They charge us to run databases on SSD but did not configure PostgreSQL properly. After contacting them we got way better execution plans: We moved a huge table to another database. We have to keep audits of every change in the system by law and this requirement is implemented with gem PaperTrail. This library creates a table in the production database where all changes of objects under monitoring are saved. We use library multiverse to integrate another database instance to our Rails app. By the way — it is going to be a standard feature of Rails 6. There are some configurations: Describe connection in the file config/database.yml Base class for ActiveRecord models from another database: Model which implements PaperTrail versions: Use case in the model under audit: We finally added more RAM to our PostgreSQL instance and currently we consume only 10% of available disk IO. We survived up until this point because we applied a few tricks — compressed data in our production database, corrected configuration and moved not relevant data away. Probably, these won't help in your particular case but I hope this article could give some ideas about custom and simple optimisation. Of course, don't forget to go through the check list of standard problems listed in the beginning. P.S.: Highly recommend a great DBA add-on for psql. Software developer