fabiochiu/medium-articles huggingface dataset:
FROM 0 TO CODER:
By: Joshua Perez Leduc
Tags: Postgresql, React, Nodejs, Rails, JavaScript

I’m currently using MacOS Big Sur (11.2.1), MacBook Pro 2019. Just in case you happen to be on a completely different version and you’re having issues.

First: Xcode

The first thing I would do in install Xcode. Easiest thing. Go here, links to Mac App Store: Xcode.

or sudo xcode-select --install in terminal.

Why? — CLI Tools.

Second: Github and Homebrew

Fun fact: Github Desktop makes it easier to install the cli tools and login to your account.

Go to Github Download Page.

Open terminal and enter:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)

To Install Homebrew.

Open Github Desktop and login, then click on ‘Github Desktop’ to reach the ‘Install Command Line Tool…” link.

Alternatively, you can run: brew install gh in terminal, and that will do the trick.

However, consider installing the Github Desktop… it has many benefits: Intro to GitHub Desktop

Third: Ruby & RVM (Ruby Version Manager)

Run the following command from Terminal. It will install RVM and the most recent stable version of Ruby.

\curl -sSL https://get.rvm.io | bash -s stable --ruby

Easy one line… if you want to know more about RVM or Ruby

Do not use brew to install Ruby… just because rvm and ruby work better together and I made the horrible mistake of install ruby through brew and then again through RVM and it was just… a mess.

Fourth: PostgreSQL

You have two options: Postgresapp.com or Homebrew, either option works… but choose ONE! I also installed both on this step and the ‘port in use’ error is the absolute worst. If for some reason you run into this error run this: sudo pkill -u postgres

So option 1: Postgresapp.com — latest version as of this post is 13

option 2: brew install postgresql

brew is always the easiest and arguably the best option, but you're a beginner and a lot of this is brand new to you. Luckily there is documentation for everything online: Postgres Docs | Postgres Brew

Fifth: Node.js

Now we’re getting to the good stuff. Nodejs is Javascript. No doubt the easiest thing to do is to go to this link and download the pkg file: Node

Run the pkg file and check your node version in terminal by running: node --version (as of this, latest: v14.15.5 LTS or v15.8.0 current)

This download should also install the Node Package Manager or NPM. Check that by running npm --version

You could also install Node via NVM (Node Version Manager): curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash

then run

env VERSION=`python tools/getnodeversion.py` make install DESTDIR=`nvm_version_path v$VERSION` PREFIX=""

then nvm use ##version number##

Sixth: Breathe

You’ve installed (nearly) everything you need to get started. This has been a long journey. AND if it went anything like mine did… you’ll have take big sighs and considered giving up. But if you made it this far… you’re doing great!

Seventh: Watch my newest Vlog!

So this vlog will show you how to take your recent installations and use it to create your first web app!
How to remove PostgresSQL on ubuntu 18.04
Tags: Postgresql, Ubuntu

1. sudo apt-get — purge remove postgresql

firdaus@firdaus-testing-server1:~$ sudo apt-get — purge remove postgresql

[sudo] password for firdaus:

Reading package lists… Done

Building dependency tree

Reading state information… Done

The following packages will be REMOVED:

postgresql*

0 upgraded, 0 newly installed, 1 to remove and 43 not upgraded.

After this operation, 66,6 kB disk space will be freed.

Do you want to continue? [Y/n] Y

(Reading database … 192250 files and directories currently installed.)

Removing postgresql (11+204.pgdg18.04+1) …

2. dpkg -l | grep postgres

firdaus@firdaus-testing-server1:~$ dpkg -l | grep postgres

ii pgdg-keyring 2018.2 all keyring for apt.postgresql.org

ii postgresql-11 11.5–1.pgdg18.04+1 amd64 object-relational SQL database, version 11 server

ii postgresql-client-11 11.5–1.pgdg18.04+1 amd64 front-end programs for PostgreSQL 11

ii postgresql-client-common 204.pgdg18.04+1 all manager for multiple PostgreSQL client versions

ii postgresql-common 204.pgdg18.04+1 all PostgreSQL database-cluster manager

3. sudo apt-get — purge remove pgdg-keyring postgresql*

4. sudo rm -rf /var/lib/postgresql/ #directory data postgresql

5. sudo rm -rf /var/log/postgresql/ #directory log postgresql

6. sudo rm -rf /etc/postgresql/ #directory base postgresql

7. sudo deluser postgres

8. sudo delgroup postgres
How to Run PostgreSQL and pgAdmin Using Docker
By: Mahbub Zaman
Tags: Postgresql, Data Science, Software Engineering, Docker, Programming

Setup

First, you will need to install Docker. I’ll use macOS for demonstration purposes.

Method 1

We will use a Docker compose file for our first method, and we need to put the docker-compose.yml inside a folder. In this case, the name of the folder is pgAdmin. Let’s break down the individual ingredients of the docker-compose.yml file.

version: '3.8' services:

container_name: pg_container

image: postgres

restart: always

environment:

POSTGRES_USER: root

POSTGRES_PASSWORD: root

POSTGRES_DB: test_db

ports:

- "5432:5432"

pgadmin:

container_name: pgadmin4_container

image: dpage/pgadmin4

restart: always

environment:

PGADMIN_DEFAULT_EMAIL:

PGADMIN_DEFAULT_PASSWORD: root

ports:

- "5050:80" db:container_name: pg_containerimage: postgresrestart: alwaysenvironment:POSTGRES_USER: rootPOSTGRES_PASSWORD: rootPOSTGRES_DB: test_dbports:- "5432:5432"pgadmin:container_name: pgadmin4_containerimage: dpage/pgadmin4restart: alwaysenvironment:PGADMIN_DEFAULT_EMAIL: admin@admin.com PGADMIN_DEFAULT_PASSWORD: rootports:- "5050:80"

First, we are using a version tag to define the Compose file format, which is 3.8. There are other file formats — 1, 2, 2.x, and 3.x. You can read more from Docker’s documentation.

Then we have a hash called services. Inside this, we have to define the services we want to use for our application. For our application, we have two services, db, and pgadmin.

We have used the tag container_name for both services to change the default container name to pg_container and pgadmin4_container for our convenience.

The second tag image is used to define the Docker image for the db and pgadmin service. To make our setup process quick and easy, we will use the pre-built official image of PostgreSQL and pgAdmin.

In my previous Docker posts, I’ve talked about the use of restart, environment, and ports tags. Have a look at the following post, where you can learn these tags.

Now run the following command from the same directory where the docker-compose.yml file is located.

cd pgAdmin

docker compose up

The command docker compose up starts and runs the entire app. Congratulations!, you are successfully running a PostgreSQL database and pgadmin4 on your machine using Docker. Now let’s connect pgadmin4 to our PostgreSQL database server.

First, access the pgadmin4 via your favorite web browser by vising the URL http://localhost:5050/. Use the admin@admin.com for the email address and root as the password to log in.

Create a server.

Click Servers > Create > Server to create a new server.
Devart Rolled Out New dbForge Studio for PostgreSQL v.2.3
By: Jordan Sanders
Tags: Database, Postgresql, Dbforge, Dba

Devart released a new version of dbForge Studio for PostgreSQL v.2.3. The IDE features massive improvements in its Script Generator, SQL Formatter, and Query Profiler.

Devart, a recognized vendor of professional database management software for developers and DBAs, launched dbForge Studio for PostgreSQL v2.3. The updated solution got huge improvements in its built-in script generation and formatting features, as well as an improved visual PostgreSQL query analyzer.

The new version includes the following enhancements:

Script Generator Improvements. The vendor added new options for script generation. Generate Script as: CREATE SEQUENCE, DROP\CREATE TABLE, CREATE INDEX, DROP and CREATE for all Source Objects.

The vendor added new options for script generation. Generate Script as: CREATE SEQUENCE, DROP\CREATE TABLE, CREATE INDEX, DROP and CREATE for all Source Objects. SQL Formatter Improvements. In the new version of dbForge Studio for PostgreSQL, the SQL Formatter was updated to work with the following statements: CREATE TRIGGER, CREATE INDEX, CREATE SEQUENCE, CREATE TABLE, CREATE VIEW, CREATE MATERIALIZED VIEW, PROCEDURE\FUNCTION.

In the new version of dbForge Studio for PostgreSQL, the SQL Formatter was updated to work with the following statements: CREATE TRIGGER, CREATE INDEX, CREATE SEQUENCE, CREATE TABLE, CREATE VIEW, CREATE MATERIALIZED VIEW, PROCEDURE\FUNCTION. Query Profiler Improvements. Now users can get a plan of any query without actually running it. This can help with graphical analysis and further improvement of high-cost queries.

dbForge Studio for PostgreSQL is a GUI tool for database development and management. This IDE for PostgreSQL allows users to create, develop and execute queries, edit and adjust the code to specific requirements in a convenient and user-friendly interface.

For more information about dbForge Studio for PostgreSQL, please visit https://blog.devart.com/dbforge-studio-for-postgresql-v23.html

About Devart

Devart is one of the leading developers of database tools and administration software, ALM solutions, data providers for various database servers, data integration and backup solutions. The company also implements Web and Mobile development projects.

For additional information about Devart, please visit https://www.devart.com/
Django : Adding initial data to a PostgreSQL database from external CSV files
By: Abiraj Rajendran
Tags: Postgresql, Django, Django Web Framework, Django Framework, Django Database

Photo from enterprisedb.com

The Django Web Framework has a default Object Relational Mapper (ORM) that helps us to do database operations easier with various relational databases such as MySQL,SQLite and PostgreSQL.

Here, we are going to add our data to our PostgreSQL database from a .CSV file.The .CSV file I have used is available here.Let’s get going..

STEP 1: CONFIGURING THE PROJECT

We are going to use django-extensions in this project.You can have it using pip

pip install django-extensions

Then, we are creating a django project named myproject and inside it, an app named myapp.Along with the folders ‘myproject’ and ‘myapp’ (in the base myproject folder), create a folder named ‘scripts’ with two .py files inside it.

__init__.py (keep this file empty,it is required to make Python treat directories containing the file as packages) load.py (this contains our code to write the data to the database)

(Refer below…)

django-admin startproject myproject cd myproject python manage.py startapp myapp mkdir scripts

Project Structure

Using pgAdmin ,create a new database named ‘sales’.

Then add our app, django-extensions to myproject/settings.py file.

Since we are using PostgreSQL database here, we have to change the default database setting from SQLite to PostgreSQL.

Paste the ‘100 Sales Records.csv’ file in the scripts folder and rename it to ‘100_Sales_Records.csv’ (to avoid file name conflicts).

STEP 2 : MODELS AND MIGRATIONS

Our ‘100_Sales_Records.csv’ file has 11 columns.We take the first 4 columns and create our Model which helps in creating the table.

In the myapp/models.py, we add the below code.

Creating the model

After adding the above,we now need to make migrations ,which in turn creates our table(as myapp_records) in the ‘sales’ database.(Django automatically adds unique ‘id’ value for each row in the table).

Making the migrations

STEP 3: load.py AND runscript

Add the following code in the ‘scripts/load.py’ file.

However, you can use pandas too, for the csv operations.(if you copy your csv file directory and paste it, make sure to change it from ‘\’ to ‘/’)

Records.objects.all().delete() deletes the previous values present in the table.

Now,run the following command

python manage.py runscript load

(runscript command from the ‘django_extensions’ is used to execute the load.py).You can see the rows being written to the table.

Now, we can see the values from our csv file in the database table.

SUMMARY

Now we can use the written values to build data to the app that you build.In my case, I am gonna use it to build a REST API using the pretty-friendly Django Rest Framework and its Serializer classes.

Thanks for reading !.

Reference : https://www.youtube.com/watch?v=E8y_6S2Fc5s
PostgreSQL to Pandas using Python
By: Neda Peyrone
Tags: Postgresql, Pandas Dataframe, Psycopg2, Pandas, Python

I’m working on a Python project that needs to read data from a PostgreSQL table. So, I want to create a class for database (DB) connection. Why’s that? Because it makes code much easier to read and reusable. Furthermore, I want to convert SQL query result into a pandas dataframe. The benefit of using pandas, it provides an easy way to manipulate data. Okay, Let’s do it! First, I install psycopg2-binary and pandas via pip. Then, create a wrapper class to deal with DB connection and fetch data, called DBConnection class.

This class consists of 4 functions as follows:

1) open_connection is used to create a DB connection.

2) disconnect is used to close a current cursor object and destroy a DB connection.

3) fetch is used to retrieve data from a SQL query.

4) execute is used to manipulate and populate DB (e.g., CRUD operations).

Demonstrating how to write DBConnection class in Python.

To convert SQL query result into a pandas dataframe using DBConnection class, follow the code below.

conn = DBConnection(DB_HOST,

DB_NAME,

DB_USER,

DB_PASSWD,

DB_PORT,

DB_SCHEMA) records = conn.fetch('SELECT * FROM SAMPLE')

column_names = ["id", "name"]

df = pd.DataFrame(records, columns=column_names)

df.head()

Happy Coding!
Update an existing user table after comparing it with new user list on Postgresql database.
By: Himanshu Pratap
Tags: Postgresql, Dba, System Administration

Update an existing user table after comparing it with new user list on Postgresql database. Himanshu Pratap Jan 8·2 min read

Objective

Update existing user table based upon new user list received such that

a. Deactivate users who are not in new list but were old table.

b. Update password of users as per new list.

c. Insert users records who are present only in new list.

We have a user table with fields :

Existing users table is as follows:

id - bigint auoincrement sequence_generator

login - varchar

password_hash - varchar

activated - boolean

say for example the existing user table data is as follows -

id,login,password_hash,activated

1,Ram,pass1,true

2,Shyam,pass2,true

3,Mohan,pass3,true

4,Sohan,pass4,true

new userlist is as follows-

login,new_password_hash

Ram,pass1

Mohan,newpass3

Sohan,pass4

Sita,pass5

Gita,pass6

After updation the user table should look like this -

id,login,password_hash,activated

1,Ram,pass1,true

2,Shyam,pass2,false

3,Mohan,newpass3,true

4,Sohan,pass4,true

5,Sita,pass5,true

6,Gita,pass6,true

Solution

I have used Postgresql-11 database with table users whose data need to be updated.



1. Create a newtable newusers

CREATE TABLE newusers ( login varchar(50) primary key, new_password_hash varchar(60) );

2. Upload data to the newusers table

\copy newusers (login,new_password_hash) from ‘newlist.csv’ delimiter ‘,’ csv header;

3. Get number of users which are present in old list but not in new list

select count(*) from users where login not in (select login from newusers);



4. Deactivate the users which are not present in the new list.

update users set activated = false where login not in (select login from newusers );

5. Verify if all users who were not in newlist has been deactivated.

select count(*) from users where activated = true and login not in (select login from newusers);

6. Count users which are present in both list.

select count(*) from users where login in (select login from newusers);

7. Update password of users which are present in both list

update users set password_hash = newusers.new_password_hash from newusers where users.login=newusers.login;

8. Count users which are present only in the new list

select count(login) from newusers where login not in ( select login from users);

9. create users that are present only in new list

insert into users( id, login, password_hash,activated)

select nextval(‘sequence_generator’) as id, login, new_password_hash as password_hash, true::boolean as activated from newusers where newusers.login not in ( select login from users);

10. Verify that new users are created.
How to Create and Connect to a PostgreSQL Database?
Tags: Database, Postgresql, Sql, Web Development, China

From PostgreSQL’s Official Website

What is PostgreSQL?

PostgreSQL is older but very powerful open source relational database.

PostgreSQL supports most of the SQL standards and provides many other modern features such as complex queries, foreign keys, triggers, views, transaction integrity, and so on. Generally it is loved by developers of startups around the world.

21YunBox provides a fully managed (free of operations and maintenance) PostgreSQL database.

How to Create a PostgreSQL Database in China?

Register as 21YunBox member Select “Database” in the console Then click “Add Database” Create with the following configuration

Note: Database name and Database user name cannot be changed once created.

You will find that in the above configuration, the database password is not filled in. This is because the system will automatically generate it and no additional filling is required.

Create a PostgreSQL database showing a configuration screenshot

If you are not sure how to do it, you can watch this example video.

How to Remotely Connect to the Database?

In most cases, we connect to the database to: add, delete, modify or search and other actions that are connected by code. We have automatically spliced ​​the URL connected to the database, to complete the configuration, you only need to paste it .

How to Connect to the Database Under the Intranet?

Under the intranet of 21YunBox, you can directly use database URL to connect.

For details, please refer to the database connection section:

Python Flask or Django project, please refer to How to deploy Django project to the server?

PHP Laravel project, please refer to How to deploy Laravel project to server?

Screenshot of PostgreSQL after creation

How to connect to a Database Under the External Network?

Under 21YunBox’s external network, you can use the client visual interface tools such as DBeaver, or Postico, or directly use command line to connect.

If you are not sure what to do, please watch the following video:

Use DBeaver to connect to the database: Use the command line to connect to the database:

How to Backup the Database?

21YunBox provides 7 days of free cloud database backup, which is rolled daily and stored in a multi-backup cloud disk in .dump format.

An example of a PostgreSQL backup file after creation

21YunBox is a Cloud Service Platform based out of Beijing, China. Our services provides you with an easy and budget friendly, end-to-end web platform deployment service within China for all applications. No matter your stack, we have a flexible solution to assist. If needed, we also offer full support in obtaining your ICP license which allows you to legally host online content within Mainland China.

To learn about the differences between 21YunBox and these foreign (outside of China) Cloud Service Platforms? You can refer to:
Django limitation with PostgreSQL < 11 for a large table
By: Md. Al-Amin
Tags: Database, Postgresql, Python, Django

To give a little bit of context, my team is building/managing an online food delivery platform. One of the largest food delivery platform in Bangladesh.

A few months ago, We had to add two new fields to our Order table in order to incorporate a new feature. Easy, right?

We did the following —

Add new fields to the Order Model Run makemigrations to generate the migration file. Make necessary changes in the serializers and views Write tests for new fields and functionalities QA engineers run tests in Dev/QA staging servers

All went well and we were ready to push the code in production. We didn’t have CI/CD integrated back then. Our production process was also simplified —

Build a Docker image Push to our own hosted docker registry Update deployment version in Kubernetes Run migration with Kubernetes Job / Exec to the container and run command

When we run the migration commands and waiting to see it completed. But wait it’s not completed yet even after 10 minutes. With millions of rows in the order table, we thought it might take a little bit more time to complete.

After 30 minutes, It’s not completed yet. Our Postgres DB went to the lock state, due to too many requests at that moment and was trying to access the DB. We had to rollback and even restart the DB.

At that time, we couldn’t realize the issue. After doing some research, we figured out that there is a caveat with PostgreSQL and Django

A caveat with PostgreSQL and Django

Adding columns with default values causes a full rewrite of the table, for a time proportional to its size. For this reason, it’s recommended you always create new columns with, null=True , in this way they will be added immediately.

We fixed our model and migration with null=True in the new columns. It took only two minutes to run the migration this time.

Tips: It’s possible to create a custom migration to add a column with null=True and change it later with another migration if you don’t want to keep the column as a null=True field.
Installing Postgres
By: Brian Mayrose
Tags: Postgresql, Development, Postgres, Tutorial, Ubuntu

Installing Postgres

Setting Up The Postgresql Database

Install Postgres

If you are using ubuntu 16.04 this is very easy to set up. First thing is to update your system:

sudo apt-get update

Now use apt-get to install Postgres:

sudo apt-get install postgresql postgresql-contrib

Check that Postgres is installed by opening another terminal and switch to the default Postgres user:

sudo -i -u postgres

Then log into Postgres:

psql

You can also access Postgres without switching accounts using this command:

sudo -u postgres psql

While in the Postgres terminal, create a password for the Postgres user:

\password postgres

Create a new user

While logged in as the Postgres’ default user you can create new users and adjust their roles.

postgres@satalite2:~$createuser --interactive

Or if you did not switch accounts:

sudo -u postgres createuser --interactive

Database GUI

I found DBeaver is the easiest to install on Ubuntu here is the link:

Once DBeaver installs select Postgres. Then connect to your database using the password you created.

Host is localhost and database and User are set to postgres
Getting Started with PostgreSQL on Mac OSX and Django
By: Kaivalya Yadav
Tags: Postgresql, OS X, Django, Mac

Install Homebrew

You can install homebrew by running the following command onMac Terminal

$ /usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

This will create folders in local machine to accommodate Homebrew’s storage requirements.

2. Update Homebrew Dependencies:

$ brew update

This may take some time

3. Install PostgreSQL:

$ brew install postgresql

4. Create Postgresql Database:

$ initdb /usr/local/var/postgres

5. Manually Start Postgres Database Server:

$ pg_ctl -D /usr/local/var/postgres start

6. Configuring Postgres:

$ psql postgres

(You may need to use sudo psql postgres )

This will open postgres shell, postgres=#

7. Adding a User

postgres=# CREATE USER username WITH PASSWORD ‘password’;

8. Create a Database

postgres=# CREATE DATABASE dbname;

9. Grant Access to User

postgres=# GRANT ALL PRIVILEGES ON DATABASE dbname to username;

10. Install the psycopg2 PostgreSQL adapter for Python by running this command in terminal in your project directory

$ pip install psycopg2-binary==2.8.4

11. Update INSTALLED_APPS in settings.py

INSTALLED_APPS = [ # …

‘django.contrib.postgres’,

]

12. Edit the settings.py File of Your Django Project

Modify the DATABASES setting as:

DATABASES = {

‘default’ : {

‘ENGINE’: ‘django.db.backends.postgresql’,

‘NAME’: ‘dbname’,

‘USER’: ‘username’,

‘PASSWORD’: ‘password’

}}

13. Apply All Database Migrations

$ python manage.py migrate

14 .You can now run the development server

$ python manage.py runserver

At the end of your work you can stop postgresql server using the following:
INDEXING
By: Deepak Kumar Singh
Tags: Indexing Tips, Postgresql, Indexing

INDEXING

WHEN TO CHOOSE DIFFERENT INDEXING?

When To choose different Indexes?

1.When to choose BTREE?

B-tree indexes are suitable for columns with better selection properties (value of n_distinct is very big, or =-1), or there is a high proportion of unique values.

2.When to choose GIN?

Contrary to BTREE indexes, GIN indexes are more efficient if the columns have poor selection properties. In addition, as GIN indexes are inverted indexes, they are very appropriate for queries on combinations of multi-value elements, e.g. array, full-text search, token, etc. As interfaces for GIN indexes are open, you can customize your GIN indexes according to the individual characteristics of your data. It supports more data types, e.g. similarity between images, text similarity, etc.

3.When to choose GIST?

GIST is a generalized index interface belonging to PostgreSQL. It is appropriate for various data types, particularly heterogeneous types, e.g. geometry index, spatial index, range index, etc.

4.When to choose HASH?

Hash indexes are useful if you are only dealing with equality queries, and the length of the indexed columns is very long, more than 1/3 of the database block length. PostgreSQL 10 hash indexes will generate WAL. To ensure reliability, it also supports stream replication. Hash indexes are not recommended for PostgreSQL versions earlier than version 10, as rebuilding is required after a crash and prior versions do not support stream replication.

5.When to choose BRIN?

BRIN indexes can be used when the data and stack storage are linearly correlated. A BRIN index is a block range index. It stores the atomic information (max value, min value, average value, null ratio, COUNT, etc.) of each data block (or consecutive data blocks).It is particularly suitable for range scans.
Running pgBench on YugaByte DB 1.3
By: Franck Pachot
Tags: Yugabytedb, Postgresql, Pgbench

Did you hear about YugaByteDB, a distributed database with an architecture similar to Google Spanner, using PostgreSQL as the query layer?

I started to follow when I’ve heard that Bryn Llewellyn, famous PL/SQL and EBR product manager, left Oracle to be their developer advocate. And YugaByteDB got more attention recently when announcing that their product license is now 100% Open Source.

I like to learn new things by trying and troubleshooting rather than reading the documentation. Probably because there’s more to learn aside of the documentation path. And also because troubleshooting is fun. So, one of the great features is that the query layer is compatible with PostgreSQL. Then I’ll try to run pgBench on YugaByteDB.

It is important to mention here that I’m running all nodes in a single lab VM, so performance is not representative. And I’m testing the YSQL query layer which is still in beta. The goal is to discover and learn about the distributed database challenges, rather than evaluating the product.

Install YugaByteDB

Nothing is easier than the installation of YugaByteDB, all documented:

I install it in RHEL 7.6 (Actually OEL 7.6 as I’m running my lab in an Oracle Cloud VM). The install is just an un-tar followed by the post-install which patches the binaries so that we don’t have to set LD_LIBRARY_PATH. I set PATH to the YugaByte bin directory:



export PATH=$PATH:$PWD/yugabyte-1.3.0.0/bin

post_install.sh wget -O- https://downloads.yugabyte.com/yugabyte-1.3.0.0-linux.tar.gz | tar -zxvf -export PATH=$PATH:$PWD/yugabyte-1.3.0.0/binpost_install.sh

I create a 3 nodes cluster on this host:

yb-ctl --rf 3 create

yb-ctl status

Those nodes are created as 127.0.0.1 to 127.0.0.3 here. I access remotely and I tunnel the interesting ports with my ssh_config file:

Host yb

HostName 130.61.59.66

User opc

ForwardX11 yes

DynamicForward 8080

#YSQL JDBC

LocalForward 15433 127.0.0.1:5433

LocalForward 25433 127.0.0.2:5433

LocalForward 35433 127.0.0.3:5433

#YCQL API

LocalForward 19042 127.0.0.1:9042

LocalForward 29042 127.0.0.2:9042

LocalForward 39042 127.0.0.3:9042

# Web UI

LocalForward 17000 127.0.0.1:7000

LocalForward 27000 127.0.0.2:7000

LocalForward 37000 127.0.0.3:7000

I create a database. The “psql” equivalent here is “ysqlsh”:

ysqlsh

\timing on

drop database if exists franck;

create database franck;

\q

ysqlsh (11.2)

Install PgBench

As I tunneled the 5433 port I can use pgBench from my laptop:

pgbench --host localhost --port 15433 --username postgres franck

But the full postgres installation is also there in ./yugabyte-1.3.0.0/postgres

then, I add this in my path:

export PATH=$PATH:$PWD/yugabyte-1.3.0.0/bin:$PWD/yugabyte-1.3.0.0/postgres/bin

Initialize pgBench

I run the initialization:
Host PostgeSQL database on cloud free Part: 1
By: Ajinkya Deshmukh
Tags: Postgresql, Heroku

In this article we are going to see how to host PostgreSQL database on Heroku for free. PostgreSQL is a powerful, open source object-relational database system. PostgreSQL comes with many features, it is open source and highly extensible.

Heroku database starter tier provides expected up time of 99.5% and is well suited for development and hobby applications

Heroku hobby tier has following limitations

· Row limits of 10,000 rows

· Maximum storage of 1 GB

· Maximum connection of 20 connections

Let’s take a look at how we can make account on Heroku and deploy PostgreSQL

Step 1: Make Heroku account and create your first app

Click here to create account on Heroku, After you have created your account and login you will see Heroku dashboard screen as below

click on New button and create new app

After creating app you will see below screen

Step 2: Install add-on Heroku PostgreSQL

Click on Resources and search for Heroku Postgres and hit enter you will see below screen

click on provision and you will see Heroku Postgres enabled as below

click on Heroku Postgres and this will redirect you to Heroku data page similar as below

Step 3: Connecting to your database using pgAdmin4

Go to settings and click on view credentials, you will find

Host, Database name, Username, Password, Port

open pgAdmin4 on your local machine

and create new server

after connecting to server you will see list of database as below

now search your database name in the list of database and use your free PostgreSQL Database in pgAdmin4

In next article we will going to see how to host PostgreSQL Database to AWS cloud platform.
could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket “/tmp/.s.PGSQL.5432”?
By: Pushpa Raj B.
Tags: Postgresql, Terminal, Macos, Servers, Sockets

could not connect to server: No such file or directory Is the server running locally and accepting connections on Unix domain socket “/tmp/.s.PGSQL.5432”? Pushpa Raj B. Jun 23, 2020·1 min read

After force shutdown my laptop, reboot it and start my rails server, I get could not connect to server.

could not connect to server: No such file or directory

Is the server running locally and accepting

connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

After googling for a while, I found the root cause is because I have force shutdown my laptop, postgresql is unable to remove from the filesystem.

/usr/local/var/postgres/postmaster.pid

So as you might have guessed already, easier solution for this problem is to remove this file manually.

rm /usr/local/var/postgres/postmaster.pid

NOTE: I have installed postgresql with homebrew, you might have it installed different way so you need to find the postgresql installation directory and remove that file from there.

And restarting postgresql mostly solves my problem. Hope this is helpful for you too!! 😃
Can we predict drunk case?
By: Ezgi Nazman
Tags: Unbalance, Postgresql, Traffic Fatalities, Ensemble Learning, Feature Engineering

Can we predict drunk case?

The Fatality Analysis Reporting System (FARS) contains data on a census of fatal traffic crashes within the 50 States, the District of Columbia, and Puerto Rico. To be included in FARS, a crash must involve a motor vehicle traveling on a traffic way customarily open to the public and must result in the death of an occupant of a vehicle or a non-occupant within 30 days of the crash.

Drunk(1) and Non-drunk(0) drivers distribution

https://www.kaggle.com/nhtsa/2015-traffic-fatalities/notebooks. There are 17 tables of traffic fatalities in 2015. I considered features in accident and person tables that can be exploratory on the drunk prediction.

Features could be exploratory on the drunk prediction

Texas and Florida have the highest number of traffic crashes in 2015.

Which machine learning method should I prefer?

Binary Logistic Regression, Support Vector Machine, Gradient Boosting, Bagging Decision Tree, Random Forest, Extra Tree, AdaBoost, Stochastic Gradient Boosting, Voting Ensemble, or XGBoost?

Ensemble methods such as Bagging and boosting algorithms are learning algorithms that construct a set of classifiers and then classify new data points by taking a weighted of their predictions. However, test accuracy and F1 rates are too low since the group label is unbalanced.

Test accuracy and F1 results

What about ADASYN Sampling? Let’s see the results after Adasyn sampling.

ROC curve

CV accuracy and F1 results

According to the ROC curve and cross-validation results, the Extra Tree classifier has the best performance. Let’s check the confusion matrix.

Confusion matrices of classification results

It is clear that the Extra Tree classifier model should be preferred for the drunk prediction.
Mac 유저를 위한 PostgreSQL 설치 및 유저,데이터베이스 설정하기
By: Taeseong Park
Tags: Macos, Postgresql, Mac

EDB를 통한 설치 및 삭제

PostgreSQL을 EDB 페이지를 통해 Installer를 다운받고 설치를 했었다.

설치 후 postgre(psql) command도 유효하지 않았고, /Library/PostgreSQL/12/bin 경로에 있길래 거기에 직접 들어가서 접속 시도를 해 보았다.

그런데 로그인을 하려는데 자꾸 비밀번호가 안 맞는지 접속이 안 되었다.. 내가 패키지로 설치할 때 설정한 비밀번호로 분명 입력했는데…

/Library/PostgreSQL/12/bin$ ./psql -U postgre Password for user postgre:

psql: error: could not connect to server: FATAL: password authentication failed for user "postgre"

그래서 EDB로 설치한 PostgreSQL을 지우기로 했다. 아래 글이 많은 도움이 되었다.

stackoverflow.com: PosgreSQL 완벽히 지우는 방법

brew를 통한 설치 및 세팅

설치

$ brew install postgresql

서비스 시작

$ brew services start postgresql

이 포스팅 에서는, 컴퓨터 재부팅 때마다 서비스를 켜 주는 커맨드가 소개되고 있는데,

$ pg_ctl -D /usr/local/var/postgres start && brew services start postgresql

이 커맨드를 안 실행하고 한번 재부팅 해 보았는데 postgres 서비스가 잘 살아있는 것 같다.

psql 접속 및 데이터베이스 생성

$ psql postgres

postgres=# CREATE DATABASE <databaseName>;

사용자 생성

Postgres doesn’t actually directly manage users or groups, like most standard permission models do. Instead, it directly manages what it calls roles. (reference 문서)

Role을 새로 생성하고, 거기에 권한을 부여하는 방식으로 진행한다.

postgres=# CREATE ROLE test WITH LOGIN PASSWORD 'test!';

CREATE ROLE

현존하는 Role 확인하기

SELECT rolname FROM pg_roles;

권한 부여하기

postgres=# ALTER ROLE test CREATEDB;

지원되는 다른 option모음

SUPERUSER | NOSUPERUSER | CREATEDB | NOCREATEDB | CREATEROLE | NOCREATEROLE | INHERIT | NOINHERIT | LOGIN | NOLOGIN | REPLICATION | NOREPLICATION | BYPASSRLS | NOBYPASSRLS, …

사용자 권한 확인하기

사용자 리스트 보기

$ psql postgres

postgres=# \du

List of roles

Role name | Attributes | Member of

-----------+------------------------------------------------------------+-----------

geoseong | Superuser, Create role, Create DB, Replication, Bypass RLS | {}

test | Create DB | {}

추가한 사용자로 접속하기

$ psql postgres -U test

psql (12.3)

Type "help" for help. postgres=>

보면 postgres=# 가 아니라 postgres=> 로 바뀐 것을 확인 할 수 있다.

가 아니라 로 바뀐 것을 확인 할 수 있다. # 는 superuser 라는 뜻이고, > 는 superuser가 아니라는 것이다

데이터베이스 추가하기

postgres=> CREATE DATABASE test;

CREATE DATABASE

특정 유저에게 DB의 모든 권한 부여하기

부여 가능한 권한 모음

SELECT | INSERT | UPDATE | DELETE | TRUNCATE | REFERENCES | TRIGGER | CREATE | CONNECT | TEMPORARY | EXECUTE | USAGE

ALL PRIVILEGES: 위에 열거된 모든 권한 포함

GRANT ALL PRIVILEGES ON DATABASE [database] TO [role(user)];

postgres=> GRANT ALL PRIVILEGES ON DATABASE test TO test;

GRANT

postgres=> \connect test

You are now connected to database "test" as user "test".

데이터베이스 관련 리스트 보는 커맨드

데이터베이스 리스트 보기

postgres=> \list

테이블 리스트 보기

postgres=> \dt

특정 database로 연결하기
A clever trick— finding duplicates
By: Karen Warmbein
Tags: Postgresql, Sql, Postgres, Data Deduplication, Deduplication

The data behind the examples

I am going to use a database table that I created called solar_system_20. Here’s some metadata about the table’s columns:

ss_name — the unique name of the Solar System object

— the unique name of the Solar System object diameter — the equatorial width of the object (in miles)

— the equatorial width of the object (in miles) classification — the type of object in the record (like star or planet)

— the type of object in the record (like star or planet) ss_location — the location of the object in the Solar System

This table has facts about the 20 largest objects in the Solar System. I’ve queried this table multiple times for various SQL tutorials:

These tutorials show hands-on examples of aggregate functions, subqueries, aliases, and the GROUP BY and HAVING commands in SQL.

Let’s view this table with a simple SELECT query to obtain the first five records.

Ok, now we have a visual understanding of this table.

Finding duplicates

An important tool to have in you SQL toolbox is to be able to identify duplicates in a table. This can be as simple as looking for records that have the same name or id. Or, it can be more complex, like below.

Suppose we want to find the classifications in solar_system_20 that have more than one record. For example, with this query we can find all the classifications and records.

SELECT classification

, count(*)

FROM solar_system_20

GROUP BY classification

Result:

We can use an aggregate function with HAVING to limit the output and find duplicate classifications — meaning, the classifications with more than one record.

SELECT classification

, count(*)

FROM solar_system_20

GROUP BY classification

HAVING count(*) >= 2;

Result:

Concatenate Strings

By the way, another cool and unique tool is the ability to concatenate strings in the table. Take a look at this.

SELECT

CONCAT_WS(' -- ', ss_name, classification) ss_info

FROM solar_system_20

ORDER BY classification;

Result:
Deploy PGAdmin4 to Google Cloud Run
By: Nathan Kwabena Fletcher
Tags: Docker, Postgresql, Google Cloud Platform, Pgadmin 4

-1. Assuming you already have a Google Cloud Project… if not, do this first https://cloud.google.com/resource-manager/docs/creating-managing-projects

0. Also assuming you have installed the Google Cloud SDK… if not do this next https://cloud.google.com/sdk/docs/downloads-interactive#mac

Create a folder called “pgadmin4”

(you can still name it anything you want. It wouldn't affect the deployment) Create a file called “Dockerfile” and put the following content in it

3. Go to the “pgadmin4” folder we created using your terminal

4. Run these in the terminal

## REPLACE ‘google-cloud-project-name’ WITH THE GOOGLE CLOUD PROJECT NAME YOU WANT TO DEPLOY IT TO

gcloud builds submit — tag gcr.io/google-cloud-project-name/pgadmin4

PS: You may be required to answer a few questions about your project

gcloud run deploy — image gcr.io/google-cloud-project-name/pgadmin4 — platform managed

PS: You may be required to answer more questions about which region in the world the deployment would be hosted

THAT’S IT!

You’ll see the link to your deployment in the terminal. If not, go to the Google Cloud Run section of the Google Cloud console and explore the console.

PS: THIS IS A DEPLOYED CONTAINER SO YOUR SERVER CONNECTION CONFIGS WILL NOT BE SAVED!!!

I’ve not figured out how to do this yet. You can stop reading now, unless you want to check out the advanced options

ADVANCED OPTIONS

If you need to add/change Environment variables, you can do that from the Google Cloud Run web console.

You can also check out more documentation on pgadmin4 config you can make in the links below
A Volunteer Business : PostgreSQL
By: Esin Seçil Yilmaz
Tags: Data, Postgresql, Rdms, Database

Today, I will introduce PostgreSQL, The first part of my article series that will consist of three parts. Due to the low number of Turkish resources about PostgreSQL, I will prepare my series in both Turkish and English. If you want to read my article in Turkish, you can click the LINK. In this first article, I will write the topics “What is PostgreSQL, What are its features, what are the advantages / disadvantages of this query language?”

PostgreSQL Logo

PostgreSQL is an open source and completely free object relational database system. The story of the birth of PostgreSQL begins in 1986. Born in 1986 at the University of California at Berkeley as part of the POSTGRES project, PostgreSQL has been continuing its development for nearly 30 years.

The most important change in terms of database in PostgreSQL, which has been introduced many versions before, occurred in the 1996 version of PostQUEL when it switched to SQL. Thus, the PostQUEL database system has been revised with its current name “PostgreSQL”. PostgreSQL is one of the most widely accepted database management systems in the industry today. Because it offers users the advantages of successful data architecture, data accuracy, powerful feature set, open source. PostgreSQL is supported by many major operating systems such as UNIX, Linux, MacOS and Windows.

As an open source PostgreSQL, has been developed independently since 1996 and only with the efforts of volunteers. PostgreSQL is not developed by any institution or organization. Donations and collected aids are the only source of income so far, which makes PostgreSQL independent and free software.
This is how I solved problem of importing resources in Terraform with dynamic provider configuration
Tags: Postgresql, Terraform, Import, Rds, AWS

I use terraform (terraform 11) for provisioning and managing infrastructure in AWS. Several days ago as a planned task I started implementation to restore RDS Postgresql from snapshot or [1]PITR. Basically my case is simple, I need to restore DB using one of the options mentioned above and then point at new DB endpoint related resources and destroy after that corrupted(source) DB.

My first question was: How to do it directly from terraform script ? I googled and found a solution [2].

Unfortunately the solution above [2] works when the source DB shouldn't be deleted after restore action. So that is not my case.. Let’s move on! After searching for a while I found the following issue on Github:

Ok, I wanted to check if it is relevant for me. After the import operation I received about ~30 errors like the once described in github issue. Really sad news… If you read github issue more carefully you will find comment [3] and link [4] which helped me a lot.

So after reading [4] I started changing my project.

Before :

main.tf

provider "aws" {

region= "${var.region}"

version = ">=2.42.0"

}

provider "aws"{

region = "us-east-1"

alias = "us-east-1"

}

provider "archive" {version = "~> 1.2"}

provider "null" {version = "~> 2.1"}

provider "random" {version = "~> 2.1"}

provider "template" {version = "~> 2.1"}

provider "github" {version = "~>2.1"}

provider "local" {version = "~>1.2"}

provider "postgresql" {version = "~>1.0"} module "acm_cert_provision" {

source = "modules/acm_certificate"

omit other parameters...

}

acm.tf part of module acm_cert_provision

provider "aws"{

region = "${var.region}"

alias = "us-east-1"

}

provider "cloudflare" {

email = "${var.cloudflare_email}"

version = "~> 1.14"

}

After:

provider "aws" {

region = "${var.region}"

version = ">=2.42.0"

}

provider "aws"{

#version = "=2.29"

region = "us-east-1"

alias = "us-east-1"

}

provider "archive" {

version = "~> 1.2"

}

provider "null" {

version = "~> 2.1"

}

provider "random" {

version = "~> 2.1"

}

provider "template" {

version = "~> 2.1"

}

provider "github" {

version = "~>2.1"

organization = "${var.repository_owner}"

}

provider "local" {

version = "~>1.2"

}

provider "postgresql" {version = "~>1.0"}

provider "cloudflare" {

email = "${var.cloudflare_email}"

version = "~> 1.0"

}

module "acm_cert_provision" {

source = "modules/acm_certificate"

providers = {

cloudflare = "cloudflare"

aws = "aws.us-east-1"

}

omit other parameters...

}

acm.tf part of module acm_cert_provision

provider "aws"{}

provider "cloudflare"{}

So as you can see I moved provider configuration from descendant modules to ancestor it is main.tf file.

Now terraform import works !

Hopefully this topic will be useful. Cheers!
PostgreSQL: measuring query activity (WAL size generated, shared buffer reads, filesystem reads,…)
By: Franck Pachot
Tags: Postgresql, Wal, Performance

When I want to know if my application scales, I need to understand the work done by my queries. No need to run a huge amount of data from many concurrent threads. If I can get the relevant statistics behind a single unit-test, then I can infer how it will scale. For example, reading millions of pages to fetch a few rows will cause shared buffer contention. Or generating dozens of megabytes of WAL for a small update will wait on disk, and penalize the backup RTO, or the replication gap.

I’ll show some examples. From pgsql I’ll collect the statistics (which are cumulative from the start if the instance) before:

select *,pg_current_wal_lsn() from pg_stat_database where datname=current_database() \gset

and calculate the difference to show the delta:

select blks_hit-:blks_hit"blk hit",blks_read-:blks_read"blk read",tup_inserted-:tup_inserted"ins",tup_updated-:tup_updated"upd",tup_deleted-:tup_deleted"del",tup_returned-:tup_returned"tup ret",tup_fetched-:tup_fetched"tup fch",xact_commit-:xact_commit"commit",xact_rollback-:xact_rollback"rbk",pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(),:'pg_current_wal_lsn')) "WAL",pg_size_pretty(temp_bytes-:temp_bytes)"temp" from pg_stat_database where datname=current_database();

Most of the statistics come from pg_stat_database. The WAL size is calculated from the latest WAL write pointer exposed with pg_current_wal_lsn() and size calculated with pg_wal_lsn_diff(). I use \gset to get them as plsql substitution variables before and used them to get the difference after.

Better with examples, I create a DEMO database and a DEMO table:

demo=# create table DEMO(n int primary key,flag char,text varchar(1000)); CREATE TABLE

Insert 1000 rows:

I’ve run the following insert:

insert into DEMO

select generate_series, 'N',lpad('x',1000,'x') from generate_series(1,10000);

INSERT 0 10000

But I’ve run my queries before and after in order to get the statistics:
JS.weekly() => #57: Human Readable JavaScript
Tags: Database, Postgresql, Development, Programming, JavaScript

JS.weekly() => #57: Human Readable JavaScript

“For a long time, humans needed to “speak” like machines in order to communicate with them. And that’s still true, we still have the need for people who work in assembly and other low-level languages. But for many of us, these complexities are abstracted away. Our job, is to focus on what is readable for humans and let the machines interpret our code.”

“As it is now in 2019, given the database versions mentioned above, unfortunately, there is a significant difference between COUNT(*) and COUNT(1) in PostgreSQL. Luckily (and this is rare in SQL), all the other dialects don’t care and thus, consistently using COUNT(*) , rather than COUNT(1) is a slightly better choice for ALL measured database products from this article.”

“A service mesh allows you to connect, secure, control, and observe services in an orchestrator platform. The term “service mesh” itself applies either to the set of overlapping network connections between services in a distributed application or to a set of tools used to manage that group of connected services.”