fabiochiu/medium-articles huggingface dataset:
Planet-Centric systems mapping
By: Carolina Faria
Tags: Systems Thinking, Circulareconomy, Design Thinking, Sustainability, Service Design

At Vincit, we are developing our planet-centric design offer through which we want to help our clients build sustainable digital products and services. This tool is part of what we are building.

Project development, whether it is design-, business- or technology-driven, it is a common practice to map stakeholders. As the projects that we develop get more complex, so do the networks of stakeholders, organisations and organisms associated with them. Therefore, for complex problems and projects, we should be mapping the whole systems where they are integrated, in order to have a meaningful impact.

Powers of Ten is a tool that helps us to consider the different levels in which a problem is integrated, having a more clear idea of the scale of problems. Therefore, we based our systems map on it. This tool originates from the documentary Powers of Ten, released in 1977, by Charles and Ray Eames.

Powers of Ten (1977)

As a tool, it is used to help people think in scale. A very simple example is that it can be used in ideation, by adding constraints to brainstorming sessions, that change the magnitude of the solution space:

What if 1 person uses it? What if a million people use it?

What if it was physically larger than this room? Smaller than a deck of cards? Had no physical presence?

What if it took more than 4 hours to complete the experience? Less than 30 seconds?

But wait, why should we think of scale?

Not understanding scale is a big setback nowadays, as the problems that we face are dependent on the relations between very big and very small threats and opportunities. We can’t really understand and navigate this complexity if we don’t understand the basics, such as:

What size is everything?

What am I actually dealing with?

What is the relationship between the size of a one thing and the other?

Powers of Ten helps us navigate this complexity.

So, what is scale?

Scale is the relative size of things. We measure the sizes of things with units of measurement like a meter or a pixel. In mathematics, a power of 10 is ten multiplied by itself a certain number of times.

Increasing and decreasing scale involves linear or exponential growth or decline. In linear growth, change happens constantly. In exponential growth, not only things are changing, but the rate that they are changing is changing.

A good example of exponential growth was given by Vera Rubin, who found the first evidence of dark matter:

If you ask people how long it will take to count from one to a million, people don’t really have an idea, but it is four to five days if we counted continually. Having this number in mind, how long do you think it takes to count to a billion? People generally say ten days, but it’s actually something like twenty years.

So we are really good at understanding linear growth but not so much exponential growth. The tricky thing is that most natural and societal phenomena are exponential.
It’s Time to Start Questioning All Systems, Everywhere
By: Alexandra Suchman
Tags: Systems Change, Systemic Change, Systems Thinking, Problem Solving

Image by Alexas_Fotos from Pixabay

If there is one common theme connecting all the terrifying, life-altering, and tragic events of 2020 thus far, it might be the prevalence of bad systems. All around us, we are seeing that the institutions and structures that surround us and shape every aspect of our lives are insufficient (at best) and dangerous (at worst).

Remember the toilet paper shortages in the early weeks of the COVID pandemic? The problem was exacerbated by an overly specialized toilet paper industry that prized efficiency over functionality.

The never-ending instances of overly forceful police responses toward demonstrations against unjustly violent, racist, reactionary police practices makes clear police agencies value enforcing order over protecting public safety.

I could go on, and on, on….

How to Evaluate a System

In this time of widespread, concurrent, and intersectional crises, we see the need for system change everywhere. But systems are large, complex, and often at least partially invisible.

You can’t change a system until you can fully understand it. You need to think about:

Who created the system?

Who is the system intended to serve?

What purpose does the system serve?

How do the actual effects of the system compare with the intended outcomes?

Different Kinds of Bad Systems

There are different kinds of bad systems, and systems can be bad for different reasons.

In the case of the toilet paper shortages, the systemic failure was unintentional: in creating totally separate systems for residential and commercial toilet paper, the industry had no way to quickly redistribute supply, so millions of boxes of paper sat unused while grocery shelves remained empty and the public panicked.

I’m willing to assume that the overall purpose of the toilet paper industry is to provide toilet paper for all residential, commercial, and public restrooms: “the people” are everyone who uses toilet paper. The shortage was an unintended consequence of trying to cut costs and automate toilet paper sales and distribution. This bug in the system can (and hopefully will) be fixed by enabling more compatibility and coordination between the distributions and sales channels for residential products and those for commercial products.

In the policing example, however, the system was doing exactly as it was designed to do. The laws, regulations, and practices were created by predominantly middle- and upper-class white men to protect the communities from which these men came. Although touted as a system for “public safety,” the definitions of what constitutes a public threat and who is worthy of protection that lie at the heart of our policing systems were intentionally set up to be malicious and divisive (if you haven’t yet, please watch the documentary 13th on Netflix).

When a system is set up on problematic premise, there is no bug to fix. The system must be dismantled and a new one must be created that rejects dangerous and immoral premises.

See the difference?
Read and Output Postgresql Data Using Node.js
By: Clark Johnson
Tags: Postgres, Postgresql, Nodejs

Set up the Node.js Application

Setting up a Node.js server to access data from a Postgresql database is simple.

Start with the basic Node.js setup. We’ll create an app called myDatabaseApp.

mkdir myDatabaseApp

cd myDatabaseApp

yarn init

Since this application will use Postgresql, we’ll need the appropriate node module.

yarn add pg

Create index.js. The code will connect to a database called node_test, and output records from the users table.

const pg = require('pg') const config = {

database: 'node_test'

} const pool = new pg.Pool(config); pool.connect((err, client, done) => {

if (err) throw err;

client.query('SELECT name, age FROM users', (err, res) => {

if (err)

console.log(err.stack);

else {

console.log(res.rows);

}

pool.end()

})

})

Set Up The Postgresql Database

We’ll need a Postgresql database and a little sample data. I used Postico to quickly set this up. Postico is a very useful Postgres client for database management. It can be downloaded and installed directly from the Postico website.

Once the client is downloaded, create a new database called node_test, add a new table called users, and then add a few columns: id, name, and age.
Take your SQL from Good to Great: Part 3
By: Taylor Brownlow
Tags: Data Science, Postgres, Data, Data Analysis, Sql

Take your SQL from Good to Great: Part 3

This is part 3 of a 4-part series on some of my most valued SQL ‘hacks’. Check out the others in the series:

Part 1: Common Table Expressions

Part 2: All about those dates

Part 3: The other JOINs (You are here 📍)

Part 4: Window Functions

Have a suggestion for an under-appreciated SQL hack that’s made the difference for you? Let me know!

The Humble JOIN

JOINs aren’t anyone’s favorite part of SQL. They aren’t as flashy as window functions or as transformational as CTEs. And yet, they’re arguably our most powerful tool to access and transform relational data.

So before getting into my favorite ways to use JOINs, I thought I’d remind myself why we should be paying more attention to the long-overlooked JOIN.

1. Optimization Power

SQL queries don’t run in the order in which they’re written (or read). In fact, the SELECT statement is one of the last steps that’s completed when a query is run.

The first part of the query that’s run is the FROM/JOIN clause, meaning if we want to improve query performance (and who doesn’t), we should look first at our old friend the JOIN.

Photo by Julia Evans.

2. Creative Problem Solving

The other, and less technical reason I appreciate JOINs is that they allow us to think about our data analysis far more creatively. They act like studs on our lego blocks, allowing us to build something far greater than our database of individual tables could do on their own.

This kind of power and flexibility is one of the main reasons we still use relational databases today, despite them being nearly 50 years old.

The Default JOIN

Despite the plethora of JOIN types available to us, most people default to the LEFT JOIN. And there are some very valid reasons for that. We (in the Western world) read from left to right, it’s easier for us to conceptualize adding columns from the second table we mention, and it tends to be less computationally expensive.

But there are scenarios where chaining together a string of LEFT JOINS might be more effort, and more computationally expensive than you can afford. For those scenarios, we can look to some of the more overlooked JOIN types:
Backup and restore PostgreSQL database in a minute
By: Shoaib Mehedi
Tags: Postgres, Programming, Development, Database, Postgresql

Keep your data safe

Backup and restore PostgreSQL database in a minute

Photo by Frank R on Unsplash

PostgreSQL is a widely used database over the world. Working with databases always needs some basic features and backup/restore database is a regularly needed scenario. PostgreSQL has its own tool called pgAdmin. pgAdmin is the most popular and feature-rich Open Source administration and development platform for PostgreSQL. pgAdmin is a powerful tool for managing PostgreSQL. Some other tools to manage databases are datagrip(a product of JetBrains) is paid. Another popular one is omniDB and it is completely free. pgAdmin is also free and I suggest it when you need some features like backup/restore.

Photo by Jordan Harrison on Unsplash

With PostgreSQL, we can take a full, incremental, or continuous backup easily. You can use pg_dump utility to back up the PostgreSQL database.

Let's begin backup the database

The backup process can happen in many ways, some prefer to do with the command line, some prefer other tools but I will suggest doing it with pgAdmin.

Here is an interface of pgAdmin. Just open the project and find your desired database then right-click on it and go to the backup.

Backup options

Enter a valid filename. You have options to backup with SQL format or backup format. I prefer backup format. In the format, section must choose custom.

Backup process

The backup process has been started and it will take a while. In my case it takes up to 5 minutes, sometimes for a small database, it can take around 1/2 minutes. Finally, you got a backup file into your selected directory.

Time to restore

Restore is nothing complex but we should remember one thing, create a fresh copy of the database or erase the database otherwise the backup will not restore. Just select your database where you want to restore and right-click then select restore.

Restoring options

Just choose the format then choose the filename from the directory you saved before and hit restore. Then the process will be begun to restore the database and it will take a few minutes.

That's the easiest way I followed before.

Conclusion

Obviously, you may not need all of these in your day-to-day life. But this may help you if you are a regular user of PostgreSQL.

Have a great day!
Deploy a Flask SQLAlchemy app to Heroku Postgres
Tags: Postgres, Heroku, Flask, Sqlalchemy, Deployment

db.create_all() seemed to work, but I have no tables in pgAdmin! If your database models are stored in a separate file, import them into app.py after db is initialised. It should look like this: app = Flask(__name__)

...

db = SQLAlchemy(app)

...

from app.models import *

Send data to Postgres

Run the app and submit some data through the form (or create some data in your table). In pgAdmin, right-click the table you just created and refresh, then right-click and view/edit data.

Could not send data to server: Socket is not connected (0x00002749/10057) could not send SSL negotiation packet: Socket is not connected (0x00002749/10057) Disconnect and reconnect server (right-click on server and click Disconnect Server, then Connect Server) and refresh table again. If that didn’t work, right-click on server and click Properties, go to the Connection tab and change the Host name/address from localhost to 127.0.0.1 . Then repeat step 1.

Create Heroku app

Since I prefer using Heroku directly with Github instead of the CLI, I’m not using git init here, but note that if your code is not already a Github repo you would need to use it.

In the command line:

heroku login

# If you have the Heroku CLI installed correctly, this will open a tab in the browser and log you in to Heroku.



heroku create unique-app-name

# (replace `unique-app-name` with your own name). This creates an app on the Heroku server, and it's the place your code is cloned to so that it can be run. Check that it has appeared in your [Heroku dashboard](https://dashboard.heroku.com/).



heroku addons:create heroku-postgresql:hobby-dev -a unique-app-name

# This creates an empty postgres database for your app. Heroku uses its own database, not your pgAdmin database.



heroku config -a unique-app-name

# The url you set earlier as the `SQLALCHEMY_DATABASE_URI` points to the database you created in pgAdmin, but since your app is now going to be using Heroku's database you need a new url. Copy the DATABASE_URL that is returned.

2. Go back to the code where you set app.config[‘SQLALCHEMY_DATABASE_URI’] earlier, and insert the new url inside the else block. It should look more or less like this:

if ENV == 'dev':

app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://postgres:123456@localhost/lexus'

else:

app.config['SQLALCHEMY_DATABASE_URI'] = 'postgres://urlyoujustgotfromherokuozftkndvkayeyc:691196bfb1b1ca8318b733935b10c97a19fd41@ec2-52-71-161-140.compute-1.amazonaws.com:5432/d3pofh2b55a2ct'

If you’re working with the tutorial’s github repo, set ENV = 'prod' , otherwise set the config to use the heroku url however you want.

3. Add the following files to your root directory if they’re not yet included in your project files:
Creating tables in SQL
By: Karen Warmbein
Tags: Postgres, Sql, Postgresql

First, let’s cover creating a table of data for a toy example. Suppose you want to recreate the black_holes table I have used in many of my SQL posts, one of which is Classifying black holes with SQL. Let’s use the same data, which I have provided again, here.

Now create a table with the data columns specified. Here, the columns are:

name — the name associated with the black hole (a record); this is recorded as TEXT data type.

— the name associated with the black hole (a record); this is recorded as data type. base_mass and power — the mass of the record, each recorded as an INT (integer). To understand this, use scientific notation: base_mass x 10^ power . The unit of mass for these black holes is in solar masses, or “the mass of the Sun”. For example, the Sombrero galaxy supermassive black hole is 1x10⁹ times the mass of the Sun.

And, we will include one more column:

num — the unique number of the record, which is recorded as an INT .

To create a table in the database, we need to create a table object with the CREATE TABLE statement. There is a simple syntax for creating a SQL Server table —

CREATE TABLE table_name (

column_name data_type [NULL|NOT NULL]

);

where data_type is what kind of data that is in the column (for example, INT , TEXT , CHAR , etc.). The [NULL|NOT NULL] option specifies if the column needs a value when a record is inserted.

If a column is defined to be NOT NULL then we cannot create a record without an actual value in the column.

then we create a record without an actual value in the column. If a column is defined to be NULL then we can create a row without an actual value in the column.

Here is an example.

CREATE TABLE black_holes (

num INT NOT NULL,

name TEXT NOT NULL,

base_mass INT NOT NULL,

power INT NOT NULL

);

Finally, we can insert the data for the four records into the table with the INSERT INTO statement.
Database Abstraction Layer
By: Hemachandra Hashan
Tags: Postgres, Backend, PHP, Server Side, Database

Photo by Campaign Creators on Unsplash

By the name of “Database Abstraction Layer” is commonly used to introduce a specific type of API. Which has ability to support multiple DBMS. As in the sense of the name ‘abstraction’, it explains that the API is not tightly coupled to one underlying database.

DBAL sits between the application and the underlying database, which interact with the database. The main benefit of DBAL is, you can change database back-end without worrying about code changes in the application. It also simplifies the database code and makes it easier to connect with database and update it.

Let say, you need to change your database back-end from PostgreSQL to Oracle? This is not an easy task. You’ll have to thoroughly go through every each of PostgreSQL specific code in your application and substitute it with the equivalent code for new database.

To avoid such difficulties, DBAL serve to decouple application logic from that used to communicate with the database. By passing all database related commands through this generalized interface, it became possible switching to some other ODBC-compliant database back-end is done in the blink of an eye. It just need to change the connection settings to replace the new database back-end.

Below diagram shows what I have explained so far.

As diagram shows, DBAL sits between the actual database and the application. Hence application does not need to know how the underlying database work.

For another instance, suppose you changed the name of the tables. You would have to go through each file and search for SQL using that table and replace the name of that table. Imagine how much time you could have saved and minimized errors if you only had to edit one file, the file that had all your SQL methods. Same thing applies to the column names also.

Those are the advantages of having a “Database Abstraction Layer” within your application.

There are several database abstraction layer options we could use in day to day application development. Those are: -

☞ PDO

☞ Doctrine DBAL

☞ ADOdb

☞ PDOModel: Database Abstraction and Helper PHP Class

☞ xCRUD Data Management System

☞ PDO Crud: Advanced PHP CRUD application
“ORDER BY” clause in Postgresql
By: Nidhi Gupta
Tags: Postgres, Database, Women In Tech, Database Development, Postgresql

“ORDER BY” clause

ORDER BY clause in SQL helps us to receive data in the specified order. We have to mention ASC or DESC with an order by clause to sort data in the specified order. If order by clause is not used with the select query, it retrieves data in

unspecified order. We can use one or more columns with an order by clause to sort data in ascending or descending order.

Syntax for ORDER BY clause

SELECT COLUMN-LIST

FROM TABLE_NAME

WHERE [CONDITION]

ORDER BY [COLUMN1][ASC/DESC],

[COLUMN2][ASC/DESC]

How ORDER BY clause works in Postgresql?

Using order by clause in Postgres helps data retrieval in specified order and order can be ascending or descending. Without using Order by clause, data always retrieve in an unspecified order. Order by always works with columns, we need to mention column_name with an order by. We can sort data by one column name or multiple column names. Suppose we have not provided any ASC or DESC with an order by in our query. By default, data will be sorted in ascending order. In this clause, the null value sorts higher than the other value sorts.

Example for Order by clause in Postgresql

Let’s create a table called customer_rating which stores data for ratings made by customers for airline. This table will have three columns customer id, airline, and rating.

CREATE TABLE CUSTOMER_RATING

(

CUSTOMERID CHARACTER VARYING(50) PRIMARY KEY NOT NULL,

AIRLINE CHARACTER VARYING(50),

RATING INTEGER

);

And let’s add some data to customer_rating.

SELECT * FROM CUSTOMER_RATING;

Example #1 Order By to sort rows with a single column. SELECT AIRLINE, CUSTOMERID, RATING FROM CUSTOMER_RATING ORDER BY CUSTOMERID ASC; Example #2 Order By to sort rows with multiple columns. SELECT AIRLINE, CUSTOMERID, RATING FROM CUSTOMER_RATING ORDER BY CUSTOMERID ASC, AIRLINE DESC; Example #3 Order By to sort rows with expressions. SELECT AIRLINE, LENGTH(AIRLINE), CUSTOMERID, RATING FROM CUSTOMER_RATING ORDER BY LENGTH(AIRLINE) DESC; Example #4 Order By to sort rows with USING clause. USING<(Sorts data in ascending order) SELECT AIRLINE, CUSTOMERID, RATING FROM CUSTOMER_RATING ORDER BY CUSTOMERID USING<; USING>(Sorts data in descending order) SELECT AIRLINE, CUSTOMERID, RATING FROM CUSTOMER_RATING ORDER BY CUSTOMERID USING>;

Conclusion

Postgresql order by clause sorts rows in ascending or descending with single or multiple columns in the query. By default, sorting is always in ascending order.

Do clap, if find useful.🙂
Creating a SQL Entity Relationship Diagram (ERD)
By: James Opacich
Tags: Postgres, Erd, Entity Relationship, Postgressql, Sql

PRE-NOTE (If you want to create an ERD fast then read this)

When I initially tried to create an ERD using pgAdmin 4 I clicked on the “New ERD Project” option in the Tools dropdown and it gave me a blank grid to work off of. Because of that I actually had to manually create each table, row, and constraint in the diagram and it took me hours. The next time I attempted to create an ERD I first right-clicked on the database I wanted to work with and instead chose “Generate ERD” and initially it still appeared that I had a blank ERD to start with but when I zoomed out I noticed that an ERD was already created for me and it included all of the accurate table names, row names, constraints and relationships. In the end what had previously taken me hours to complete now only took me 30 seconds.

That being said, I found it very useful to create my own ERD manually. It got me thinking about the process, the data types, constraints, and relationships in a way that I felt further solidified my knowledge of SQL.

PREREQUISITES

This tutorial assumes you already have PostgreSQL, pgAdmin 4, and The Pagila Database installed and set-up. If you need instructions on setting this up please visit my previous article: Simulating A Real-world SQL Environment.

Please have pgAdmin 4 open at this point and your database ready to use.
Python資料庫學習筆記(六)：SQLite3
By: Yanwei Liu
Tags: MySQL, Postgres, Python, Database

建立資料庫

import sqlite3 con = sqlite3.connect('mydatabase.db')

建立資料庫Cursor變數

con = sqlite3.connect('mydatabase.db') cursorObj = con.cursor()

建立Table

employees (id, name, salary, department, position, hireDate)

使用 DB browser for sqlite 檢查資料是否建立成功

插入一筆資料到Table

cursorObj.execute("INSERT INTO employees VALUES(1, 'John', 700, 'HR', 'Manager', '2017-01-04')") con.commit() #VALUES裡面的值，可用?代替，表示缺失的空值

更新資料到Table

import sqlite3 con = sqlite3.connect('mydatabase.db') def sql_update(con): cursorObj = con.cursor() cursorObj.execute('UPDATE employees SET name = "Rogers" where id = 2') con.commit() sql_update(con)

選擇資料

cursorObj.execute('SELECT * FROM employees ')

取得所有資料

print(row) for row in cursorObj.fetchall()

取得指定資料

import sqlite3 con = sqlite3.connect('mydatabase.db') def sql_fetch(con): cursorObj = con.cursor() cursorObj.execute('SELECT id, name FROM employees WHERE salary > 800.0') rows = cursorObj.fetchall() for row in rows: print(row) sql_fetch(con)

顯示有多少ROW在上一次操作時受到影響

rows = cursorObj.fetchall()

print len (rows)

print(cursorObj.execute('DELETE FROM employees').rowcount)

列出所有Tables

import sqlite3

con = sqlite3.connect('mydatabase.db')

def sql_fetch(con): cursorObj = con.cursor()



cursorObj.execute('SELECT name from sqlite_master where type= "table"')



print(cursorObj.fetchall())



sql_fetch(con)

檢查表格是否存在

import sqlite3 con = sqlite3.connect('mydatabase.db') def sql_fetch(con): cursorObj = con.cursor() cursorObj.execute('create table if not exists projects(id integer, name text)') con.commit() sql_fetch(con)

刪除Table

import sqlite3



con = sqlite3.connect('mydatabase.db')



def sql_fetch(con):



cursorObj = con.cursor()



cursorObj.execute('DROP table if exists employees')



con.commit()



sql_fetch(con)

插入大量資料

import sqlite3



con = sqlite3.connect('mydatabase.db')



cursorObj = con.cursor()



cursorObj.execute('create table if not exists projects(id integer, name text)')



data = [(1, "Ridesharing"), (2, "Water Purifying"), (3, "Forensics"), (4, "Botany")]



cursorObj.executemany("INSERT INTO projects VALUES(?, ?)", data)



con.commit()

關閉與資料庫的連接

con = sqlite3.connect('mydatabase.db')

con.close()

建立時間資料
How to Deploy a Django App on Heroku
By: Gustavo C. Maciel
Tags: Postgres, Django, Heroku, Programming, Cloud Computing

First of all, you need a Procfile , which is used to declare the application process type. This file must be located on your projects’ root directory. You can create one running the following command on your terminal:

echo "web: gunicorn PROJECT_NAME.wsgi" > Procfile

Where PROJECT_NAME is the name of your Django project.

Next, you need to install some packages to make the application work on the server. These are the required packages:

gunicorn (an application server)

dj-database-url (for the database)

whitenoise (for serving static files)

psycopg2 (for postgres database)

To install them all in once just go to your terminal and run:

pip3 install gunicorn dj-database-url whitenoise psycopg2-binary

After the installation has completed, add them to your requirements.txt:

pip3 freeze > requirements.txt

Now you have to create a runtime.txt file on your root directory (just like the Procfile) and add the python version you’re using. To check which python version you're using, run:

python3 --version

The output will be something like this:

Python 3.8.5

Since I’m using Python 3.8.5, to create the runtime file and add the python version I can just run the following command:

echo "python-3.8.5" > runtime.txt

You should also create a .gitignore file and add the sqlite database to it as long as the virtual environment folder. To do this run:

echo "db.sqlite3" > .gitignore

echo "env/" >> .gitignore

where env/ is the name of your virtual environment.

Note: The second command should use two arrows ( >>) instead of just one ( >), because you’re appending to the file. Using one single arrow ( >) would replace the content of the file with the new one.

Now you have to configure some of the project files to make the application work on the server. First, go to the settings.py file and do the following:

To allow whitenoise to handle the static files, you have to add the following line to the list of INSTALLED_APPS :

Add the following line to ALLOWED_HOSTS list:

You also have to set Debug to False:

Now you have to add whitenoise to the MIDDLEWARE list but this part is tricky, the whitenoise middleware should be included right after the django.middleware.security.SecurityMiddleware

Now you have to add the configuration for the static files. At the bottom of your settings.py , add these lines:

Finally, but not least, you have to configure the Postgres database

Right after the DATABASES dictionary, add the following lines:

Yeah, that was a lot, but now your app is ready.
Lessons learned running svelte locally
Tags: Zenuml, Postgres

I am migrating ZenUML web application from basing on a fork of web-maker to svelte. I am recording here afew lessons learned during this process.

PostgreSQL

I have PostgreSQL installed.

Login to PostgreSQL

$ psql postgres psql (10.2) # version of postgres

Type "help" for help. postgres=# # the prefix gives the current database name

Which database am I currently using?

postgres=# select current_database(); current_database

------------------

postgres

(1 row) # to switch database. Note that you are NOT switch db with psql. There is no way to switch db in postgres itself. This just closes one connect and open another. postgres=# \c db_zenuml

You are now connected to database "db_zenuml" as user "pengxiao".

db_zenuml=# \connect db_zenuml

You are now connected to database "db_zenuml" as user "pengxiao".

function gen_random_uuid() does not exist

# !!! make sure you are using the correct database

db_zenuml=# CREATE EXTENSION IF NOT EXISTS pgcrypto;

pgAdmin

Set password for user postgres

postgres=# \du # list all users

List of roles

Role name | Attributes | Member of

-----------+---------------------------------------------------

pengxiao | Superuser, Create role, Create DB | {}

postgres | Superuser, Create role, Create DB, ... | {}

zenuml | Create DB | {} postgres=# ALTER USER postgres WITH PASSWORD 'password';

Download pgAdmiin from https://www.postgresql.org/download/ and install it. When you create a new server connection, you need the postgres user and its password. You can use other users as well.

DB migrate

When I run npm run migrate it always failed at the beginning. The migrate script is "migrate": "node-pg-migrate -r dotenv/confg" . I think they missed an up before -r . Note that -r dotenv/config may not be necessary. You should create a .env file with the following configuration:
How to create and connect to a Postgres container using docker-compose and .env files
By: Anderson Loiola
Tags: Postgres, Docker Compose, Docker

How to create and connect to a Postgres container using docker-compose and .env files Anderson Loiola Feb 16·5 min read

Today we are going to discuss how to create and connect to a Postgres database using Docker and docker-compose.

First and foremost to start our project, we will need to install Docker, and to do that, you can just follow the tutorial in the links in the references section.

To create our database with docker, we will create a Docker-compose.yml that will be responsible to instruct Docker to pull our desirable docker image (in our case the Postgres docker image) from the Dockerhub and do some configuration like mapping ports to expose the psql DB to the host machine alongside a database.env file that will contain the configuration of the database.

With that in mind let’s get started.

Step 1 - Create a Docker-compose.yml file:

Your Docker-compose will have to look like the following image:

docker-compose.yml config file

As you can see, at the start of the file we are instructing the docker that we are using the 3.0 version, next, with the container_name property we are naming our container attributing the postgres_db name to him, next in the image property we are telling Docker that we are using the Postgres db docker image, with the port attribute we are telling respectively that we are ‘binding’ the 5432 docker container port to the 7000 host port, and lastly but not least with the env_file we are telling Docker to load the environment variables contained in our database.env file in our docker image.

Step 2 - Creating the database.env file:

To run the Postgres Docker image properly, we need to provide some env variables, and with a .env file, we can easily group those variables to increase the maintainability and decrease the complexity of the configuration.

database.env config file

Basically what we are doing here is describing our environment variables in a key: value syntax where the key, represents the name of the variable that we are going to create and the value describes its value.

3 - Executing the docker-compose:

Now that we have all the configuration ready we can execute de docker-compose up command to create our container by just typing docker-compose up in a terminal that’s executing in the same folder as our docker-compose.yml file.

docker-compose up command

To see if our container is running we can type the docker container ls command in our terminal and then it will list all the running containers.
Elixir: Ecto — find records on empty associations
By: Minh Reigen
Tags: Postgres, Elixir, Ecto, Association, Sql

In my experience and the apps I worked on, there were a few times I needed to find records based on empty associations.

Let’s imagine a silly example where you were writing a software to manage your franchise of electronic stores. You wanted to find store locations that run out of both Sega Genesis consoles and VCR players (I know, I’m old.) Being a programming nerd and recently picked up on Elixir, you would want to write your own Ecto query to find those. How would you do this?

NOT EXISTS with fragment function is the answer! (Well, you can technically do this with LEFT JOIN and NULL check too, but let’s go with NOT EXISTS in this post)

You can also combine another condition where the stores also have no delivery truck.

I know this is not in the scope of this short post, but I can’t help. You can use defmacro to actually clean this up a bit:

I hope you get the idea of using NOT EXISTS and apply it in your scenarios when you need to use it.

Please check out my other Elixir posts if you are interested. Thank you!
How do download and installation of PostgreSql Admin
By: Ponnumani Gurusamy
Tags: Postgres, Database, Postgresql

First We do Start download in PostgreSql Below Mentioned Link

https://www.enterprisedb.com/postgres-tutorials/how-quickly-build-api-using-nodejs-postgresql

Then Select Downloads Button on Top Corner on Right Side.

Then Open Next Page of EDB Account. Select Postgres SQL’s Right Arrow

Then Open PostgreSQL Download Page. Select Window x86–64 Download Button.

Then Open EDB Account Page. Enter Below Mentioned Details for Create EDB Account

· First Name & Last Name

· E-Mail id & Business Phone

· Company Name & Organization Type

· Country

· User Name

Then Click Agreement Check Box and Select Create an Account Button.

Then Showing Dialogue Box for Checking Mail. EDB account Activation Confirmation mail has sent to User Mail-id. Then Link has Sent through Mail to User.

Click Below Mentioned Link for Change EDB Account Password.

https://www.enterprisedb.com/user/reset/680767/1612854906/pgOOqtGx-MoPMmvxztLPrqynO1sFp6MAiKzWgIXWZq4/login?redirectTo=https://www.enterprisedb.com/user/register?destination=/thank-you-page?cid=437&action=activation

Then Open Setup-PostgresSQL Box. Then Select Next Button.

Then Open Installation Directory and Check installation Directory. Then Select Next Button

Then Open Setup Component Box. Select All Components Then Next Button When You are ready to Continue.

Then Open Data Directory Box and then Check the Directory Details. Then Click Next Button

Then Open Existing Data Directory Box and then Select Next Button. The Directory its Configuration will be used for this Installation.

Then Open Pre Installation Summary then Showing Instruction of Setting used for Installation. Then Select Next Button.

Then the Process ready to Installation and Select Next Button.

When The Installation Process is Completed and Select Next Button.

When Completed PostgresSQL Process and the Select Finish Button. Click the Check Box is used for Download and Installation Application Tools, Drivers. Then Click Finish Button.

Then click home button and select Postgre SQL.

Click PgAdmin and it should open in the browser. After that enter the password..

Then Open Double Click on Servers in Left Side Server Tab.
Building Spring boot Rest API using Apache Camel and PostgreSQL.
By: Supun Kavinda
Tags: Postgres, Java, Spring Boot, Apache Camel, Camel

3. Add PostgreSQL configuration and Create Classes.

Before dealing with code let’s configure our application properties in the application.properties file located in the source resource directory.

application.properties file

Note: Please change the database user name, password, and port according to your configuration. If you are not specified camel.component.servlet.mapping.contextPath value. Then it’s default value is camel.

Now We can start codding. As a first step, we can add a Book Entity class to our project. I am going to add the Book entity class under the entity package. That code will look like below.

Book.java class

The next step is creating Repository interface for the book entity. So we can create a BookRepository interface by extending JpaRepository interface. That class creates an inside respository package. It’s like below.

Then I am going to create BookService class to handle the CRUD operation. This class create inside service package.

4. Create camel routes for basic crud operations.

Now we come to the final stage of this sample project. until now we create all the required configuration and classes except the rest controller class. also, we didn't use Apache camel in any place. So I am going to create a rest controller class using camel Java DSL without using spring MVC. If we create a rest controller using camel’s route, we have to extend our rest controller class using RouteBuilder. Once we extend the rest controller from RouteBuilder we have to override configure method. We define all our routes in the configuration method like below.

Here, I create four routes to do CRUD operation. Below I include CURL request for all routes and include the postman collection in the Github with a sample project. You can use one of these methods to test this sample example.

Save Book CURL request



"id":0,

"name":"Camel in Action",

"author":"Claus Ibsen and Jonathan Anstey",

"price": 56

}' ' curl -XPOST -H "Content-type: application/json" -d '{"id":0,"name":"Camel in Action","author":"Claus Ibsen and Jonathan Anstey","price": 56}' ' http://localhost:8080/rest/book'

2. Find all Books CURL request

3. Find Book By Name CURL request

4. Remove Book By bookId CURL request

Now, this sample project implementation is complete. Before running this application we need to run the PostgreSQL server. Here I am using the PostgreSQL docker version.

docker run --ulimit memlock=-1:-1 -it --rm=true --memory-swappiness=0 --name postgresql-camel -e POSTGRES_USER=user -e POSTGRES_PASSWORD=password -e POSTGRES_DB=cameldb -p 5432:5432 postgres:latest

If you want to know more about camel you can read camel official documentation and Camel in Action book.

Here I have only implemented GET, POST, and DELETE methods. You can try to implement PUT, and PATCH methods also.

You can find complete code in my GitHub account.

If you have any questions please put a comment I am happy to help you.
Querying By Timestamps
By: Nurudeen Ibrahim
Tags: Postgres, Ruby on Rails, Timestamp, Activerecord

Recently I came across this strange behavior with querying by created_at field in Rails/Postgres. I was working around two tables; versions and updates.

Version.last.created_at

=> Tue, 05 Feb 2019 20:08:31 UTC +00:00 Update.last.created_at

=> Tue, 05 Feb 2019 20:08:31 UTC +00:00

Then I tried to query the versions table by the update’s created_at.

Version.where(created_at: Update.last.created_at)

=> []

Giphy.com

I was surprised by this result because I was actually expecting the above query to return an array containing the last version. This prompted me to do another check.

Version.last.created_at == Update.last.created_at

=> false

Giphy.com

I decided to make some research, then I realized that doing a direct comparison with timestamps does not always give expected results, and the best way to ensure an accurate result is giving is by casting them to integer. Then I did:

Version.last.created_at.to_i == Update.last.created_at.to_i

=> true

Yeah true!

In order to achieve what I wanted, I discovered a Postgres CAST method which helps in types casting. So I ended up with something similar to this:

Version.where("CAST(extract(epoch from created_at) AS INT) = ?", Update.last.created_at.to_i) => {id: 5800287, created_at: Tue, 05 Feb 2019 20:08:31 UTC +00:00, ...}
Postgres: dynamic functions execution
By: Valery Yakovlev
Tags: Postgres, Sql

In this tutorial I will explain how to perform dynamic function call, pass parameters to that function from table rows and fetch result.

This may be useful, for example, when you store function name as text in database and have to call it while iterating on rows.

Let’s define several functions: multiplication and division:

CREATE FUNCTION mul(numeric, numeric) RETURNS numeric AS 'select $1*$2;' LANGUAGE SQL IMMUTABLE RETURNS NULL ON NULL INPUT;

Division function(div) is already defined and it performs division of integers.

So in order to perform division of numerics I had to create another function “mydiv”:

CREATE FUNCTION mydiv(numeric, numeric) RETURNS numeric AS 'select $1::numeric/$2;' LANGUAGE SQL IMMUTABLE RETURNS NULL ON NULL INPUT;

Let’s define table which will contain two numeric operands for operations and the operator name. Operator name corresponds to PLSQL function defined above:

create table test_operator ( value1 numeric not null, value2 numeric not null, operator_fun text not null );

Now let’s insert some data. This data contains numeric parameters and operation name:

insert into public.test_operator (value1, value2, operator_fun) values (666.666, 2.0, 'mul'); insert into public.test_operator (value1, value2, operator_fun) values (666.666, 2.0, 'mydiv');

Wrapper function which will execute passed query and return result set. I don’t have solution without it yet, so it seems to be mandatory:

CREATE OR REPLACE FUNCTION public.myexec(text) RETURNS SETOF RECORD LANGUAGE 'plpgsql' AS $BODY$ BEGIN RETURN QUERY EXECUTE $1 ; END $BODY$;

And the final query! Now we can run command which will iterate our rows, call specified function with parameters and calculate result:

select tt.value1, tt.value2, tt.operator_fun, t.res from public.test_operator tt, myexec('select ' || tt.operator_fun || '(' || tt.value1 || ', '|| tt.value2 || ')') as t(res numeric)

Calculation results are printed in column “res”.
Extending Postgraphile with External API Requests
By: Aaron Case
Tags: JavaScript, Postgraphile, Postgres, GraphQL

With that in mind, let’s break it down step by step.

makeExtendSchemaPlugin is our bread and butter, and gql allows us to write graphQL syntax right in a JS file. I’m using node-fetch here for simplicity’s sake. This is a backend application, and thus there is no access to window, and fetch won’t play nice.

This is where we name our new types. In this case we know the fields that come back from our external API call, so we map them all here. Next, we extend our query with a field name, and our new type.

Finally, we write our resolver. Using our field name from above, we pair it with a function call. In this case we reach out to our external API and fetch some data.

All that’s left to do here is fire up your Postgraphile application. In this use case, it’s done via a command on server start:

In the options parameter, we simply pass our new plugin to the appendPlugins key. As you can see, it’s an array that can take in any number of extensions.

Once this is done, you can see it in action in a graphiQL sandbox.

I hope this helps you make some great stuff!
Use PostgreSQL instead of Elasticsearch when you don’t really need it
By: Corentin Vilain
Tags: Postgres, Data, Development, Elasticsearch, Full Text Search

Using the power of PostgreSQL with Full Text Search combined with Similarity

What does PostgreSQL offer us?

The pg_trgm extension:

The pg_trgm module provides functions and operators to determine the similarity of alphanumeric text based on trigram matching, as well as index operator classes that support fast searching for similar strings.

CREATE EXTENSION pg_trgm;

Full Text Search:

Full Text Searching (or just text search) provides the capability to identify natural-language documents that satisfy a query.

I won’t go into details here but here is the basic usage:

select id, name from event

where to_tsvector('english', name) @@ phraseto_tsquery('market:*'); 34 | Preparation of the marketing

67 | Online market launch

The goal is to bring up results that are lexically close to our search. We can see it as a conventional “LIKE ‘%my_search%’” but at the grammatical level. (The real advantage is that it is much more effective than a traditional “LIKE” but we see that in the next part.)

Similarity:

select * from similarity('market', 'marketing'); 1 | 0.54

similarity(text, text) returns a number that indicates how similar the two arguments are. Knowing that the closer the number is to 1, the more the two arguments match.

You can use the ‘%’ operator that will bring up searches that have a similarity higher than 0.3 (by default):

select id, name, similarity(name, 'market') from event

where (name % 'market'); 12 | Market | 1

13 | Mark | 0.5

45 | Mars | 0.42

57 | Black Market | 0.38

You can change the default similarity value with:

SET pg_trgm.similarity_threshold = 0.2 ALTER DATABASE db SET pg_trgm.similarity_threshold FROM CURRENT; --To persist the configuration and not just for the session

The combination of text search and similarity:

select id, name from event WHERE (to_tsvector(name) @@ phraseto_tsquery('market:*')) or ('market' % search); 34 | Preparation of the marketing

67 | Online market launch

12 | Market

13 | Mark

45 | Mars

57 | Black Market

The goal here is to combine the two queries to add fuzzyness to our search. Specifically, we will be able to retrieve results that are grammatically close and similar. We will see right after that it will be very useful to combine the two queries to determine the order and the accuracy of the results.
Building Your Own Facial Recognition System
By: Ted Troxell
Tags: Machine Learning, Facial Recognition, Postgres, Computer Vision, AI

Prerequisites

So, before we get started, let’s go over some prerequisites for this project. Here are the things you’ll need in order to successfully build your own facial recognition system.

Understanding how facial recognition works

A T3 API Key

Python3.7+

Postgresql version 12.6

PostgREST version 7.0.1

React version 17.0.2

Node version 10.19.0

Those who just want to look at the code, here’s a link to the repo. Now, let’s develop a better understand of how facial recognition works and how we will use it.

How Does Facial Recognition Work?

For this article we won’t be diving into Convolution Neural Networks, Harris Corners or other technical means of the mechanical inter-workings of these systems. Instead, we want to focus on how these systems determine who is who.

In facial recognition software, the machine learning models are first trained on hundreds of thousands or millions of images that have been painstakingly annotated by hand of key facial features within a picture of someone’s face, like the image below.

source: quora.com

The points on the face above calculated by a machine learning model that “looks” at an image and does it’s best to figure out where certain facial features are located. Using this information, facial recognition systems can calculate the geometry of your face, or your facial fingerprint, if you will. Then, this fingerprint is compared against a database of fingerprints for similarity in facial features. What sets facial recognition systems apart is the number of features they use to measure your face as more features, typically mean higher accuracy.

For our system, what we’ll do is take a few pictures first to build a more accurate digital fingerprint, that way we don’t have to keep our head in the exact same spot every time we want to use it.

The Setup

As I mentioned earlier, we’ll need a T3 API Key to create our facial recognition app. T3 is an API service that I’ve created that offers free machine learning APIs to developers. For this, we’ll use the /facekp endpoint to get the key points from a face.