fabiochiu/medium-articles huggingface dataset:
Rails + Postgres + UUID
By: Dan Foley
Tags: Postgres, Rupauls Drag Race, Ruby on Rails, Uuid, Database

Overview

What the heck is UUID? Why would I want to use a UUID? Rails Setup DatabaseSetup The Big Reveal! Epilogue

1. What the heck is UUID?

A universally unique identifier (UUID) is a 128-bit number used to identify information in computer systems…While the probability that a UUID will be duplicated is not zero, it is close enough to zero to be negligible. — Wikipedia

Here’s an example:

2. Why would I want to use a UUID?

Bad actors: and no, I don’t mean Steven Seagal and his perma-sunglasses face.

Mr. Seagal prepares to brush his teeth in public.

When you’re just using the straight-outta-the-box incremental integer IDs, a bad actor may be able to intuit quite a bit about your resources. It makes URL-hacking that much easier. If Mr. Seagal visits his user account page and peeps the URL that reads:

cooldudes.com/users/13

He might see that he’s user #13 and find that to be an unlucky number and leave the site. Seagal might think a site with only 13 users is embarrassing then tweet about it. He also might also see his messages page URL:

cooldudes.com/users/13/private-messages/7

…and think to himself “Hmmm. I wonder if I can peep other people’s private messages by substituting those integer ids for other numbers that I think of while brushing my teeth.”

So yeah, we want to avoid people making too many inferences on our backend for some decent OpSec!

Mega secure with Becky Stern’s “Compubody Sock”

3. Rails Setup

Go ahead and spin up a new Rails API with a Postgres database. PG is great for Heroku deployment especially since they don’t support Rails’ default database: SQLite

rails new app-with-uuid-keys --api --database=postgresql

Before we generate migrations for any of your models you’ll want to tell Rails that when generating migrations it should use uuid instead of plain old integers by creating this file:

Next you’ll want to generate your models/migrations for your Postgres database. But first we must enable the pgcrypto extension in our Postgres database:

rails generate migration enable_uuid --no-test-framework

This migration file should end up looking like this:

When generating your models if you specifically rails generate a model and not a migration then the Rails generator config that we set up moments ago will handle the uuid-ing of the user’s primary key.

Running rails g model user name:string email:string --no-test-framework generates a blank User model class annnnnd this beautiful migration:

Users migration table

4. Database Setup

Go ahead and run rails db:create && rails db:migrate

This is what you should see!

5. The Big Reveal
How to display database content in your flask website
By: Dul S Blog
Tags: Postgres, Flask, Python, HTML, MySQL

Flask,PostgreSQL

File structure

Hi,I have a table called v_timestamp1 made in PostgreSQL. Lets see how we can display it in the Flask UI. First I m establishing connection with pgadmin in the try statement. Put your database name instead of silverTip, that is name of my database also my user is postgres, my password is sT789. Replace those with your specific credentials.Here is my app.py code

from flask import Flask, request,render_template

import psycopg2

try:

conn = psycopg2.connect(database="silverTip", user="postgres",

password="sT789", host="localhost")

print("connected")

except:

print ("I am unable to connect to the database")

mycursor =conn.cursor()

app = Flask(__name__)

def index(): @app .route('/')def index(): return render_template('index.html')

def v_timestamp():

mycursor.execute("SELECT * FROM v_timestamp1")

data = mycursor.fetchall()

return render_template('v_timestamp.html', data=data) @app .route('/v_timestamp')def v_timestamp():mycursor.execute("SELECT * FROM v_timestamp1")data = mycursor.fetchall()return render_template('v_timestamp.html', data=data)

If you guys havent downloaded psycopg2, please do so before running the code, in the terminal. Please make sure you have made a database made in the postgreSQL.

pip install psycopg2

Code in index.html.In the body section I have just included a button, upon clicking that button lead to v_timestamp page.

<!DOCTYPE html>

<html>

<head>

<title></title>

<link rel="stylesheet" href="{{ url_for('static', filename='home.css') }}">



</head>

<body> <h2>Vehicle info dashboard</h2> <p align="center"><a href=v_timestamp ><button class=grey style="height:75px;width:150px">vmd timestamp</button></a></p> </body>

</html>

The index.html has the code for arrangement of styles for the main page(app.py). How the front page looks like in port 5000.

The code in the v_timestamp.html is given below. I have given the text alignment as center, and padding and border as specified in the code.My table column headings are vehicle ID, current timestamp, expiry timestamp.As specified under <th> element, those fonts appear in bold. The <tr> elements font is normal(not bold) .In the <td> elements, the instruction {{row[0]}} means to capture the first row details in the v_timestamp1 PostgrSQL db and so on. Do not get confused, v_timestamp1 is the db at postgreSQL. v_timestamp is the page I’m routing the main page upon a button click event.

<!DOCTYPE html>

<html>

<head>

<title></title>

</head>

<body>

<style>



td {

width: 150px;

text-align: center;

border: 1px solid black;

padding: 5px;

}

</style>

<table>

<thead>

<tr>

<th>vehicle ID</th>

<th>current timestamp</th>

<th>expiry timestamp</th>

</tr>

</thead>

<tbody>

{% for row in data %}

<tr>

<td>{{row[0]}}</td>

<td>{{row[1]}}</td>

<td>{{row[2]}}</td>

</tr>

{% endfor %}

</tbody>

</table> </body>

</html>

How the UI looks for v_timestamp page.

Thanks for reading. Have fun coding!!!
How to fix an error: password authentication failed for the user in PostgreSQL
By: Hassan Al Najjar
Tags: Postgres, Data, Error Message, Database, Postgresql

I have two solutions for this Error.

First solution :

open Terminal and write this command in terminal

sudo editor /etc/postgresql/<postgresql version >/main/pg_hba.conf

you can click Tab , if you didn’t know PostgreSQL version

Then, go to the bottom of the file you just played and edit these lines

- if scroll didn’t work you can use keyboard buttons to scroll down

# TYPE DATABASE USER ADDRESS METHOD



# "local" is for Unix domain socket connections only

local all all peer

# IPv4 local connections:

host all all 127.0.0.1/32 md5

# IPv6 local connections:

host all all ::1/128 md5

you must change peer and md5 to trust to be as this

# TYPE DATABASE USER ADDRESS METHOD



# "local" is for Unix domain socket connections only

local all all trust

# IPv4 local connections:

host all all 127.0.0.1/32 trust

# IPv6 local connections:

host all all ::1/128 trust

peer means it will trust the identity (authenticity) of the UNIX users. So not asking for a password.

means it will trust the identity (authenticity) of the UNIX users. So not asking for a password. md5 means it will always ask for a password, and validate it after hashing with MD5 .

means it will always ask for a password, and validate it after hashing with . trust means it will never ask for a password, and always trust any connection.

Then follow these steps to fix Error:-

1- click Ctrl + s to save your edits.

2- click Ctrl + x to close the file.

3- run this command sudo service postgresql restart

Second solution :

If you remember correctly the user postgres has no DB password set on Ubuntu by default. That means, that you can log in to that account only by using the postgres OS user account.

Assuming, that you have root access on the box you can do:

If that fails with a database "postgres" does not exists error, then you are most likely not on a Ubuntu or Debian server :-) In this case simply add template1 to the command:

sudo -u postgres psql template1

If any of those commands fail with an error psql: FATAL: password authentication failed for user "postgres" then check the file /etc/postgresql/8.4/main/pg_hba.conf : There must be a line like this as the first non-comment line:

local all postgres ident

For newer versions of PostgreSQL ident actually might be peer . That's OK also.

Inside the psql shell you can give the DB user postgres a password:

ALTER USER postgres PASSWORD 'newPassword';

You can leave the psql shell by typing Ctrl-D or with the command \q .

Now you should be able to give pgAdmin a valid password for the DB superuser and it will be happy too. :-)
How to Resolve ERROR: could not resize shared memory segment “/PostgreSQL” No space left on device with Postgres Docker Image
By: Shanika Wickramasinghe
Tags: Postgres, Shared Memory, Resize, Docker, Wso2

How to Resolve ERROR: could not resize shared memory segment “/PostgreSQL” No space left on device with Postgres Docker Image Shanika Wickramasinghe Sep 11, 2020·2 min read

If you are using Postgres DB server with a docker image and if you want to add more users for your database there is a chance that you may encounter the error mentioned in the subject of this blog post. I will mention the quick steps that you can follow to get this resolved by resizing your shared memory allocated for the docker container.

I was able to get this resolved on top of a ubuntu 16.04 OS with the below steps.

Get the docker image container ID for the relevant postgres server

sudo docker ps -a

2. Now check the allocated shm space for that conatiner id. (a1d302620d5c is the container id)

sudo docker exec -it a1d302620d5c df -h /dev/shm

OUTPUT

Filesystem Size Used Avail Use% Mounted on

shm 64M 28K 64M 1% /dev/shm

3. Stop the docker container if its running

sudo docker stop a1d302620d5c

4. With your local machine navigate to below folder location

/var/lib/docker/containers/a1d302620d5ca267f633eea8f5d99f05b5f5c132b0b6dd13fcbf51379d24be35(container id)

5. Shut down the docker service (This is mandotory if not the changes you apply would not be available)

sudo service docker stop

6. Check the status

sudo service docker status

7. Open the file with vi tool /var/lib/docker/containers/a1d302620d5ca267f633eea8f5d99f05b5f5c132b0b6dd13fcbf51379d24be35/hostconfig.json

8. Locate the element “ShmSize”: and change it as below to allocate 1GB space

“ShmSize”:1073741824

9. save the file

:wq!

10. Start the docker service

sudo service docker start

11. Check the server status and it should show as its running.

12. Now get the docker container ID and start the docker container

sudo docker ps -a

sudo docker start a1d302620d5c

13. Now check the allocated shm space for that container ID

sudo docker exec -it a1d302620d5c df -h /dev/shm

OUTPUT

Filesystem Size Used Avail Use% Mounted on

shm 1.0G 28K 1.0G 1% /dev/shm
Monitoring PostgreSQL Databases using Postgres exporter along with Prometheus and Grafana.
By: Sylia Ch
Tags: Postgres, Prometheus, Grafana, Monitoring, Postgresql

Photos via Pexels

Introduction

In my previous articles , i have covered monitoring using Prometheus along with Grafana. and alerting using the AlertManager integrated to Prometheus.

In today’s article, i will mainly cover the topic of integrating the Postgres exporter to Prometheus in order to get PostgreSQL databases metrics for a monitoring purpose.

Setting up the Postgres exporter

Download the Postgres Exporter Binary

First, you need to download the postgres exporter binary tar archive and untar it as follows

Prepare the env File

Then, you need to prepare an env file for the postgres exporter where we will set a variable known as DATA_SOURCE_NAME.

Metrics from all defined databases in DATA_SOURCE_NAME will be collected.

cd /opt/postgres_exporter sudo nano postgres_exporter.env # Inside the postgres_exporter.env put the following: DATA_SOURCE_NAME="postgresql://username:password@localhost:5432/database-name?sslmode=disable" # or you can use the following to monitor all the databases available on localhost DATA_SOURCE_NAME="postgresql://postgres:postgres@localhost:5432/?sslmode=disable"

Setup the Postgres Exporter Service

First, create the user postgres

sudo useradd -rs /bin/false postgres

Then the service

sudo nano /etc/systemd/system/postgres_exporter.service

Next, put the following inside

[Unit]

Description=Prometheus exporter for Postgresql

Wants=network-online.target

After=network-online.target [Service]

User=postgres

Group=postgres

WorkingDirectory=/opt/postgres_exporter

EnvironmentFile=/opt/postgres_exporter/postgres_exporter.env

ExecStart=/usr/local/bin/postgres_exporter --web.listen-address=:9187 --web.telemetry-path=/metrics Restart=always [Install]

WantedBy=multi-user.target

Finally, enable and start the service

sudo systemctl daemon-reload

sudo systemctl start postgres_exporter

sudo systemctl enable postgres_exporter

sudo systemctl status postgres_exporter

Integrate the Postgres exporter to Prometheus

First, you need to edit prometheus.yml configuration file.

sudo systemctl stop prometheus

sudo nano /etc/prometheus/prometheus.yml

Then, add the new targets to scrape

— job_name: ‘postgres_exporter’

static_configs:

— targets: [‘xxx.xxx.xxx.xxx:9187’]

Finally, reload the service

sudo systemctl start prometheus

sudo systemctl status prometheus

Display the metrics on Grafana Dashboard

There is plenty of interessting Grafana Dashboard for PostgreSQL monitoring:

PostgreSQL Database (which i found the most adapted for me)

Postgres Overview

Postgres Overview

You can also use the 6742 ID

Choose the Prometheus datasource and get your result.

Grafana Dashboard

References

All databases on the eyes. Postgres exporter + Prometheus + Grafana.
Streaming Data to Big Query
By: Tpt Rich
Tags: Postgres, Golang, Google Bigquery

As you know streaming data is very needed because we need analytic, BI, training, report or do something else. But we don’t want to touch the product database. The streaming is a solution.

This part I will explain how I streaming Postgres to Big Query. I need comment pro and cons of that. Now getting start.

Requirements

Some require and use case

Performant: Changes are propagated to subscribed consumers in near real-time.

Consistent: Ordering and timeline consistency.

Flexible: We can config change source and destination.

Technologies used

Golang

gRPC

Docker

Postgres

Architecture

High-level workflow overview

Implement

gRPC we need to implement data structure streaming in proto3. Because we need to know exactly what is operation “Insert, Update, Delete …”, RawEvent and Event. Look like this!

syntax = "proto3";

package pqs;



import "github.com/golang/protobuf/ptypes/struct/struct.proto";



service PQStream {

// Listen responds with a stream of database operations.

rpc Listen (ListenRequest) returns (stream Event) {}

}



// A request to listen to database event streams.

message ListenRequest {

// if provided, this string will be used to match table names to track.

string table_regexp = 1;

}



// An operation in the database.

enum Operation {

UNKNOWN = 0;

INSERT = 1;

UPDATE = 2;

DELETE = 3;

TRUNCATE = 4;

}



// RawEvent is an internal type.

message RawEvent {

string schema = 1;

string table = 2;

Operation op = 3;

string id = 4;

google.protobuf.Struct payload = 5;

google.protobuf.Struct previous = 6;

}



// A database event.

message Event {

string schema = 1;

string table = 2;

Operation op = 3;

// if the id column exists, this will populate it

string id = 4;

// payload is a json encoded representation of the changed object.

google.protobuf.Struct payload = 5;

// changes is, in the event of op==UPDATE an RFC7386 JSON merge patch.

google.protobuf.Struct changes = 6;

}

We will declare trigger in Postgres, it will trigger to topic we defined. And will have consumers to progress.

Magic happens here. We using NotificationChannel in Postgres and Golang lib support github.com/lib/pq

Subscriber event and callback handler

I have open source, please checkout.
Heroku Data Links — Practically. The steps to making Heroku Data Links…
By: Scott Goley
Tags: Postgres, Data Link, Heroku

Creating the credential on both instances is most easily done in code:

heroku pg:credentials:create postgresql-sunny-1234 --name limited_user -a sushi

There example as referenced in the documentation here:

2. You will need to grant permissions on is most easily accomplished in pgAdmin through the grant wizard unless you already have a grant script:

In my case, I’m only granting select permissions to all the necessary tables for this credential which can be done for the whole schema at once through this tool.

3. After doing both of the previous steps, I quite frequently found myself frustrated by the following error — something along the lines of:

permission denied on foreign table 'example-table' for user powerbi-prod

error 42801

Now, this is covered fairly extensively in the FDW documentation but Heroku gives little warning. You will need to create a user mapping for each credential you mean to utilize via the following on the new local instance:

CREATE USER MAPPING FOR "read_only_user_username" SERVER postgresql_trapezoidal_12345 OPTIONS (user 'yyy', password 'xxx');

⚠️Keep in mind that the underlying credential is still governed by Heroku PG. As such it will be subject to a rotation period with every scheduled maintenance that either instance goes through. This rotation can break the mapping!️⚠

With that, you should be able to create new views or materialized views referencing the foreign tables and query those remote targets directly.

I hope to hear back from you about what use cases you come up with for your Heroku Data Links connection!

PS. As alluded to via my credential — we at Finexio utilize Heroku Data Links together with PowerBI via a custom connector to generate something I call “run-time” analytics. Hopefully, you can utilize some of these tools to do the same!
Work with AWS RDS Proxy
By: Crespo Wang
Tags: Postgres, AWS, JavaScript, Serverless, MySQL

Create a proxy

It’s pretty straightforward to create an RDS Proxy. Just a few things to remember when you create a proxy.

>VPC Security Groups. The security group assigned to the Proxy must allow traffic from your application environments, such as Lambda or EC2, to the Proxy, and the traffic from the Proxy to the RDS. If the security groups are not configured properly,

> IAM Authentication. When you choose to use IAM Authentication you must use TLS to connect to the Proxy. And when IAM Authentication is required, native credentials, ie username and password authentication will not be accepted for connection.

If the Proxy isn’t configured properly, you normally will see the target becomes unavailable, it means the connection to the RDS isn’t ready, in most cases the cause is either the security group or the credentials to the RDS.

Luckily with AWS CLI, we can debug it easily.

aws rds describe-db-proxy-targets --db-proxy-name dev-rds-proxy-test

For example, I deliberately set the wrong Secret Manager. The target becomes unavailable because of the wrong secret.

The most problematic issue can be the security group setting. When the security group is not right, it normally means the traffic isn’t allowed, so you would see the following error.

You have to be patient and find out which SG isn’t correct.

Note that there are a few seconds delay between the configuration changes and the state changes.
Image Recommendations with PyTorch + Flask + PostgreSQL + Heroku deployment
By: Maciej D. Korzec
Tags: Postgres, Machine Learning, Heroku, Recommender Systems, Python

Image Recommendations with PyTorch + Flask + PostgreSQL + Heroku deployment

Selected image of a logo and automatically detected logos of other technologies employed for the image recommender application (Source: M. D. Korzec; Flask logo license; Python logo PSF trandemark usage policy; html logo cc by 3.0)

During the last weeks I have implemented an item-to-item image recommender system that uses PyTorch’s pretrained Resnet18 convolutional neural network to generate comparability through feature vectors, a database to manage the images and top-k lists and a user interface.

The developed solution and steps for the deployment on Heroku are presented in this article and the code is available on GitHub.

I have set and fulfilled the following requirements for this project

1. The web app is accessible on the internet.

2. The application has a gallery of images with pagination, loading only the relevant data for the page.

3. The data for the gallery and the recommendations is fetched from a database.

4. The recommendations are most similar images from a possibly large image set. They must appear instantaneously; this means that the similarity calculation happens offline.

5. The gallery is responsive.

6. The Flask application is suitably packaged for future extensions/reuse.

In my last post already parts of the requirements were realized as prototype, in particular 2 and 4. A Jupyter notebook is available at GitHub for the Resnet18 PyTorch recommender that takes an image as input and gives out most similar images from the same image set. To use the Flask app this notebook should be used to generate Pandas dataframes that are stored as pickle files. These are used as static content in the app.

In this post we will show how to realize also all other requirements, how to use an SQLite database locally and PostgreSQL on Heroku, how to use it for pagination, how to clean-up the work and to deploy it to Heroku to make it available on the internet via browser on PCs or smartphones.

You can access and test the demo app with a few images on Heroku through the following link:

This post is structured in the following sections:

1. The features of the app are described

2. The calculation of similarities between images is revisited

3. The packaging structure for the application is described

4. The SQL model is presented

5. The backend application logic is introduced

6. The templates are introduced

7. All steps for the Heroku deployment are explained

8. A conclusion is made

Note, for this article I was using a Windows 10 machine and Python 3.5.4. I have left out any exception handling.
Migrating from Sqlite3 to Postrgresql in Ruby on Rails
By: Thomas Dubiel
Tags: Database, Postgres, Rails, Ruby

Often times while creating web applications, we do not give much thought to the storage method used. Sqlite3 is the default database that is shipped with Ruby on Rails, and functions well enough in the local development capacity. However, once we start thinking of deploying our applications to production, we must consider other options.

The first step in working with postgres is installing it on our system:

gem install pg

Now that we have installed postgres on our system, we will need to create a new “role” for ourselves in postgres. In the following command, we switch into the postgres user and create a new role, passing in as parameters the name of the role, which will end up being your username, and the password:

su - postgres create role stockapp with createdb login password 'password'

We now run the following command to generate a new app while specifying which database we would like to use:

rails new stockapp --database=postgresql

Obviously name your app whatever you like. The file that we must work with now is config/database.yml. We must change it to look like the following:

I have omitted the password field values as I had them hard-coded. This is not a safe habit and not recommended, and in your own application, the preferred methodology is to store your passwords in environment variables in a separate file, and reference those variables in the password field. If using Github, you will need to include that file that stores your environment variables in your .gitignore file before committing to Github.

For instance, in the config directory in Rails, you can great a file called pg_key.yml, and there we can set an environment variable like so:

PG_KEY: “password”

And in out config/application.rb file, past the following code:

Before our application loads, it will look for the pg_key.yml file and give your rails application access to the environment variables set in the file.

Now run the server:

bin/rails server

And voila! You have successfully created an application using postgresql!

Summary

Postgres is a very robust database that is used frequently in production. There is obviously quite a bit more setup here than with using the standard database that ships with Rails. However, the benefits are reaped in the future. If you are planning on pushing your application to production, do yourself a favor and use postgres for both development and production environments, as this is the preferred approach.
Python Batch algorithm for getting consecutive events(streaks) from app user data
By: Seoyeon Chang
Tags: Postgres, Data Science, Streaks, Python, Pandas

One of my recent project required to create a job that pulls in users activities with a timestamp and build the longest and the latest streaks for all users in a batch.

I ended up writing a job in python and hooked it up to an hourly worker. The job mainly does these four things in a series

1. pull in (userId, createdAt as t) records from a source table

2. load them into a dataframe and group consecutive days for each user

3. get the max and latest consecutive days for each user

4. write streaks in batch to a target table

Basically, the task pulls in these source records

and builds out this target records

Implementation is available here, I have also shared a notebook file for showing details for step #2 and #3 here

Method buid_streak_counts in the batch_streak_count.py is the task executed by an hourly job, and at the end, it writes streaks into a target table via Postgres specific batch write operation copy_from which I found ridiculously fast.

It has been in running in production for the last few weeks and has been quite robust so far.

Please feel free to comment on anything!
Rails 5, Postgres, & Windows Subsystem for Linux — An Inseparable Peace
By: Karl Mccollester
Tags: Windows Subsystem Linux, Postgres, Rails, Wsl

So my wonderful 6-year old Dell XPS 13 Developer edition’s soldering at the power port finally gave way, leaving me with a fully function laptop with only one little issue that it can’t get power. The sent me packing to my ‘backup’ Windows 10 laptop. I really wanted to keep working in a Linux environment and the new WSL with Ubuntu feature seemed like the perfect choice…. well, may be not so much. Here are some random notes I painfully learned on setting up Rails 5.2 with Postgresql 11 on Windows Subsystem for Linux: If it’s a local database, Rails will by default try to connect to Postgresql with UNIX sockets instead of through the TCP port. Per WSL issue #921, this won’t work. The solution: set host to 127.0.0.1 and include the ‘port’ in your config\database.yml file:

development:

<<: *default

host: 127.0.0.1

database: development

username: <username>

password: <password>

port: 5433

You’ll then run into the following error while attempt to create the database with rake db:migrate :

WARNING: could not flush dirty data: Function not implemented

This is discussed in WSL issue # 2628, which is a couple clicks away from this Postgresql thread on fsync. After a few wrong turns, it led me to this thread, which has you add the following line to /etc/postgresql.conf:

fsync=off

Worked like a charm… after only 6+ hours of teeth-gnashing
SQL Cheat Sheet
By: Jason Lee
Tags: Data Science, Postgres, Data, Coding, Sql

SQL Cheat Sheet

Photo by panumas nikhomkhai from Pexels

SQL is the most important coding language to learn for data analysis. Some might argue Python and R are equally important, but when it comes to the most common tool an Analyst must have it is SQL.

According to Dataquest.io almost all of the biggest names in tech use SQL. Uber, Netflix, Airbnb — the list goes on. Even within companies like Facebook, Google, and Amazon, which have built their own high-performance database systems, data teams use SQL to query data and perform analysis.

Like every language, you need to keep practicing to understand and grasp keep concepts. In my opinion, SQL is one of the easier languages to use once you understand the basic structure of the code. In this article, I share the necessary steps to getting started with SQL querying.

Standard SQL Structure

This is Part 1 to a series of PostgreSQL cheat sheets and will cover SELECT , FROM , WHERE , GROUP BY , HAVING , ORDER BY and LIMIT .

The basic structure of a query pulling results from a single table is as follows.

SELECT

COLUMN_NAME(S)

FROM

TABLE_NAME

WHERE

CONDITION

GROUP BY

COLUMN_NAME(S)

HAVING

AGGREGATE_CONDITION

ORDER BY

COLUMN_NAME

LIMIT

N

What is SQL?

SQL (pronounced “ess-que-el”) stands for Structured Query Language. SQL is used to communicate with a database. It is the standard language for relational database management systems. SQL statements are used to perform tasks such as update data on a database or retrieve data from a database.

What is Relational Database Management System (RDBMS)?

An RDBMS organizes data into tables with rows and columns. The term relational means that values within each table have a relationship with each other.

Rows — also known as records

Columns — also known as fields, have a descriptive name and specific data type.

What is PostgreSQL?

PostgreSQL is a general-purpose and relational database management system, the most advanced open-source database system.

Other common database management systems are MySQL, Oracle, IBM Db2, and MS Access.

Let’s begin!
Materialized Views
By: Komala Prabhu Tirupachur
Tags: Postgres, Materializedviews, Sql, Database, Postgresql

Consider the following scenario: you have a relational database where you store metadata for a million artists. There are also separate tables that keep track of the historical social media metric change over time for each of these artists. You want to be able to sort and/or filter artists dynamically based on their latest metrics. Will plain queries be performant enough or would a view suffice? A query using joins across all the different tables will be too slow to be run every time the artists have to be sorted by a different metric.

What are Materialized Views?

A materialized view is one such solution that can solve this problem. Views are quite commonly used in relational database systems, but the concept of a materialized view may not be familiar to many. So, what exactly is a materialized view? While a traditional view stores the definition of the query that needs to be executed upon access, a materialized view executes the query and stores the result when it is created and acts as a table for access purposes. For our example above, this would mean creating a materialized view that would store the latest social media metrics for each of the artists. This would then enable us to access the latest values and sort by any metric without having to do any heavy computation when the data is accessed.

Creating Materialized Views

Each relational database management system will have its own method of implementing materialized views. I use PostgreSQL syntax in this article.

create materialized view SAMPLE_MAT_VIEW_NAME as (<complex_select_query>);

Creating the materialized view is essentially a wrapper over the complex query whose results you want to store. The execution time of the create command will depend on the performance of the underlying select query. Once the materialized view is created, you can use select queries over it and create indexes in the same way as you would on a table.

Choosing between a traditional view vs a materialized view

With materialized views, the complex query is run only once, thus reducing the load on your database. However, this means that the data in the materialized view might become stale over a while. The data can be refreshed from time to time by using the refresh command (as shown). Using the optional concurrently keyword allows the view to be accessed even when it is being rebuilt but is only available from Postgresql 9.4. The rate of refresh can be decided by the user.

refresh materialized view concurrently SAMPLE_MAT_VIEW_NAME;

In contrast, queries run over traditional views will always return the latest data. In addition, they also have the advantage that the storage is much smaller and there is no duplication since the actual data is not stored.

Materialized views are not meant to completely replace having traditional views in your database. Each approach has its advantages, choosing between the two depends on your specific use case.
Upload csv files to Postgresql (AWS RDS) using python’s Psycopg2
By: Alejandro Madrazo
Tags: Postgres, Data Science, Computer Science, Python, Postgresql

In this tutorial I’m going to explain how to upload large cvs files from a remote location (client) to a remote postgresql server. This process is a great solution for your data science projects (ETL specifically).

PostgreSQL — Python

Create table to run our test

I suggest creating your field with varchar(n) to avoid type errors while trying to upload your CSV file.

2. Create a demo CSV file

Is important:

not having extra columns (null values are fine)

not having extra blank rows at the end

Having complex data types set in your database that do not match the incoming data. (For this I suggest (If possible) setting your fields to varchar and then doing the necessary transformations)

Example of my demo file:

Plain CSV file from a books dataset (Contains 11k+ rows)

Notice that my CVS file contains a header. (it’s ok with out it as well)

3. Preparing our code

Here you can see my clean “insert_demo” table under the business_intelligence schema.

installing and importing libraries

Next I use pandas just to verify my CSV’s shape

Here I just verified my CSV’s size. You can notice we have 11,127 rows and 3 columns

Creating our insert function

A function is not necessary but I prefer to keep everything as functions for portability and team work.

The copy function receives the following parameters:

cursor: c

file_path: “./exampleData/books.csv”

table_name: “insert_demo”

cols_tuple: tuple_demo = (“colname1”,”colname2", “colname3”,…)

Creating our variables for file path, table name and columns.

3. Function for establishing connection

Here we establish our connection using psycopg2’s connect function. We should pass to the function the following parameters:

user, password, host, port, database

Once we get the connection object we extract our cursor object which will help us to execute our commands.

Once we get our cursor, we can call our copy_query function and pass it down along with our file path, table name and columns tuple.

After calling our function which will execute psycopg2’s copy_from function. Which is the method that loads data from a remote location into postgres.

It’s important to commit our connection in order to save changes into the database permanently.

Close connection and add a exception handler

Make sure to close cursor and connection

Results:

SUCCESS!!

SUCCESS!!

AND DONE :)

Feedback is appreciated!

Happy Coding!
Always use ORDER BY when using OFFSET
By: Aaron Bruce
Tags: Postgres, Relational Databases

This is a stock photo of somebody manually reading a csv like an idiot

In the past couple of years I’ve switched from MySQL to Postgres, and this is one of the Postgres gotchas that just keeps causing problems for me.

In MySQL my default way of processing through a group of records is like so

-- Get the first batch

SELECT * FROM table LIMIT 1000 -- Get the second batch

SELECT * FROM table LIMIT 1000 OFFSET 10000

If you do that in Postgres you will skip records and you will get duplicate records. In order to prevent this behavior all you need to do is add an ORDER BY to the query, and make sure that whatever you’re using to order by is guaranteed to be unique.

-- Get the first batch

SELECT * FROM table ORDER BY id ASC LIMIT 1000 -- Get the second batch

SELECT * FROM table ORDER BY id ASC LIMIT 1000 OFFSET 1000

Problem solved!

But Why…?

The reason that happens is that Postgres is very smart about how to return records. One of the things it does to optimize performance is that if you don’t tell it how to sort records it will decide at run time what is the fastest way to sort them for you.

If Postgres decides to sort records differently from query to query, then for each batch you’re essentially getting a random set of records back from Postgres rather than a linear progression through the table.

-- Get the first batch

-- This batch might be sorted by the primary key

SELECT * FROM table LIMIT 1000

-- returns ids [1, 2, 3, 4, 5, ...] -- Get the second batch

-- This batch might be sorted by a secondary unique key

SELECT * FROM table LIMIT 1000 OFFSET 1000

-- returns ids [200, 4, 523, 1, ...] -- The second batch does not return ids [1001, 2000] which you might -- expect. It is missing some records, and it also contains some

-- duplicates from the first.

It just so happens that most of the time as you’re batching through the table the fastest sorting method is most likely going to remain the same from query to query, so if and when you see missing records and/or duplicates it will be a huge pain in the butt to find and diagnose.

So — in conclusion — the way to avoid this problem in your production applications is to ALWAYS ALWAYS tell Postgres how to sort your records (with a unique ORDER BY ) if you’re going to be using LIMIT across multiple queries to perform batch processing on records.

Or, even better, just stop using OFFSET with the seek method.
Getting binary data from PostgreSQL as a base-64 encoded string
By: Niel De Wet
Tags: Postgres, Base64

Suppose you have a bytea column containing some binary data:

When selecting, use the encode function to encode the binary data as a base-64 string:

As you can see, postgres splits the base-64 string across multiple lines, at 76 character-width. The important thing to notice here is that the + at the end of each line is NOT part of the base-64 string.

niel$ echo -n IlRoZSBxdWljayBicm93biBmb3gganVtcHMgb3ZlciB0aGUgbGF6eSBkb2ciIGlzIGFuIEVuZ2xpc2gtbGFuZ3VhZ2UgcGFuZ3JhbeKAlGEgc2VudGVuY2UgdGhhdCBjb250YWlucyBhbGwgb2YgdGhlIGxldHRlcnMgb2YgdGhlIGFscGhhYmV0LiBJdCBpcyBjb21tb25seSB1c2VkIGZvciB0b3VjaC10eXBpbmcgcHJhY3RpY2UsIHRlc3RpbmcgdHlwZXdyaXRlcnMgYW5kIGNvbXB1dGVyIGtleWJvYXJkeXBpbmcgcHJhY3RpY2UsIHRlc3RpbmcgdHlwZXdyaXRlcnMgYW5kIGNvbXB1dGVyIGtleWJvYXJkcywgZGlzcGxheWluZyBleGFtcGxlcyBvZiBmb250cywgYW5kIG90aGVyIGFwcGxpY2F0aW9ucyBpbnZvbHZpbmcgdGV4dCB3aGVyZSB0aGUgdXNlIG9mIGFsbCBsZXR0ZXJzIGluIHRoZSBhbHBoYWJldCBpcyBkZXNpcmVkLiBPd2luZyB0byBpdHMgYnJldml0eSBhbmQgY29oZXJlbmNlLCBpdCBoYXMgYmVjb21lIHdpZGVseSBrbm93bi4= | base64 --decode "The quick brown fox jumps over the lazy dog" is an English-language pangram—a sentence that contains all of the letters of the alphabet. It is commonly used for touch-typing practice, testing typewriters and computer keyboardyping practice, testing typewriters and computer keyboards, displaying examples of fonts, and other applications involving text where the use of all letters in the alphabet is desired. Owing to its brevity and coherence, it has become widely known.
PostgreSQL GROUPING SETS
By: Natarajan Santhosh
Tags: Postgres, Grouping And Sorting

How to use grouping sets in postgres

I prefer running postgres via docker

# create these directory on your home docker/volumes/postgres & docker/volumes/temp $ docker run --rm --name pg-docker -e POSTGRES_PASSWORD=docker -d -p 5432:5432 -v $HOME/docker/volumes/postgres:/var/lib/postgresql/data -v $HOME/docker/volumes/temp:/datastore postgres $ docker exec -it pg-docker bash $ root@d7a21c8e5498:/# psql -h localhost -U postgres -d postgres psql (12.1 (Debian 12.1-1.pgdg100+1))

Type "help" for help.

Create a table and use grouping sets.

postgres=# CREATE TABLE sales (

brand VARCHAR NOT NULL,

segment VARCHAR NOT NULL,

quantity INT NOT NULL,

PRIMARY KEY (brand, segment)

);



INSERT INTO sales (brand, segment, quantity)

VALUES

('ABC', 'Premium', 100),

('ABC', 'Basic', 200),

('XYZ', 'Premium', 100),

('XYZ', 'Basic', 300); postgres=# select GROUPING(brand)::boolean isbrandGrouped, GROUPING(segment)::boolean isSegmentGrouped, brand,segment, sum(quantity) from sales group by GROUPING SETS ((brand,segment), (brand), (segment), ())

;

isbrandgrouped | issegmentgrouped | brand | segment | sum

----------------+------------------+-------+---------+-----

t | t | | | 700

f | f | XYZ | Basic | 300

f | f | ABC | Premium | 100

f | f | ABC | Basic | 200

f | f | XYZ | Premium | 100

f | t | ABC | | 300

f | t | XYZ | | 400

t | f | | Basic | 500

t | f | | Premium | 200

(9 rows)

References

https://www.postgresql.org/docs/10/queries-table-expressions.html#QUERIES-GROUPING-SETS
Health workers are the frontline soldiers against COVID-19
By: Political Affairs
Tags: Covid 19, Doctors, Coronavirus, Health Workers, Covid 19 Crisis

On the frontlines as authorities around the world battle to prevent the spread of coronavirus. The COVID-19 or the new Corona Virus is different. In this virus we have an enemy which is invisible and sometimes deadly, and the task is harder. About a century ago the Spanish flu pandemic killed an estimated 50 million people, more than the combined total casualties of World Wars I and II. Our understanding of disease transmission and treatments is far ahead of our position in 1918, but this new coronavirus has shown the limits of our ability to deal with major disease outbreaks. Advice to protect ourselves is clear: wash your hands well and often, self-isolate if you feel unwell, maintain social distance by avoiding crowded and public spaces and, if your symptoms worsen, contact medical services. Only by following this advice rigorously can we hope to stem the tide of new infections.

For now, however, the virus is spreading and, on the frontline between a nervous public and those responsible for directing national responses, the healthcare workers on whom we all depend can easily be forgotten. During the Ebola outbreak six years ago, the World Health Organisation estimated that health workers were between 21 and 32 times more likely to be infected with Ebola than people in the general adult population. In West Africa more than 350 health care workers died while battling Ebola. Doctors, nurses, carers and paramedics around the world are facing an unprecedented workload in overstretched health facilities, and with no end in sight. They are working in stressful and frightening work environments, not just because the virus is little understood, but because in most settings they are under-protected, overworked and themselves vulnerable to infection.

The risk to doctors, nurses and others on the front lines has become plain: Italy has seen at least 18 doctors with coronavirus die. Spain reported that more than 3,900 health care workers have become infected, We need a whole-of-society resolve that we will not let our frontline soldiers become patients. We must do everything to support health workers who, despite their own well-founded fears, are stepping directly into COVID-19’s path to aid the afflicted and help halt the virus’s spread. In sub-Saharan Africa as elsewhere, pressure on the healthcare workforce will intensify in the coming months. A recent survey of National Nurses United (NNU) members in the US, revealed that only 30% believed their healthcare organization had sufficient inventory of personal protective equipment (PPE) for responding to a surge event. In some parts of France and Italy, hospitals have run out of masks, forcing doctors to examine and treat coronavirus patients without adequate protection.

Courtesy : Africa Renewal
DILEMMA OF A NIGERIAN DOCTOR III On Salary and Wages
By: Zahra'U Ahmed
Tags: Nigeria Health System, Doctors, Finance, Nigeria, Balanced Life

DILEMMA OF A NIGERIAN DOCTOR III

On Salary and Wages

Congratulations on your appointment doc. You finally going to start practicing as a doctor.

Parents: Baba we thank God, our son has finally gotten a job. Yes mama, the time has come that we will start reaping the fruits of our labor. At least now he can take care of some of our responsibilities.

Siblings: Bro, you remember that phone I told you I want, you said I should remind you at the end of the month.

Relatives: Uncle, school has resumed since last week. But I'm yet to go. Pls I need some money to buy books and do some shopping.

Yan unguwa: yallabai Dan Allah a taimaka, my wife just put to bed and I want to buy ram.

In practice.

Family and friends will come, you do free consultation, give them money to buy the drugs and even transport fare.

Your patients: the man is in bad shape, but cannot afford medication. You buy some for them or even help settle their bills.

You need equipment to work, you have to buy from your pocket because they are not available. Else you put yourself at risk or even watch patients die.

When you are trying to manage your life.

You wear cheap clothes: haba doc, you can't be wearing this kind of clothes. It's not up to your reputation.

Buy small car: this doc are stingy, see the type of car he's driving.

Take your children to cheap school: I wonder what this man is doing with his salary. The kids should be in a better school.

When you try to survive

Do Private practice: this doctors, see how they are jumping from one hospital to another. How ungrateful

Entrepreneurship: doc, you like money too much. Leave this kind of things for other people na..

Demand for raise in pay: chai! These doctors are so greedy. With all the money they're recieving they still demand for more.

Look for job abroad: see longer throat, always looking for greener pastures. Such unpatriotic beings.....😔😔

Dear Nigerians, we are also humans..

We have responsibilities, we want decent living with good clothes and houses too. We want the best for our kids.

We want to practice well and see the smile of our patients.

How do we manage all that, meeting to your standard demands and expectations while fulfilling all other responsibilities.

J.K Rowling is yet to give us the secret spell

First written August 28th, 2020