Hibernate N+1 and Batch Performance Issue
Tags: Spring Boot, Jpa, Hibernate

N + 1

何謂N+1問題

在系統設計上，一對多、多對多的關係十分常見，也常常需要同時抓取兩張table的內容，如果是用SQL來處理你應該可以很直覺的寫出來，

SELECT * FROM TABLEA JOIN TABLEB ON A.XXX = B.XXX WHERE CONDITION

但使用hibernate的時候，在資料量為5(N=5)的情況下，則會執行6(N+1)次 824900 nanoseconds spent preparing 6 JDBC statements; 1602200 nanoseconds spent executing 6 JDBC statements;

也就是一行SQL可以解決的問題，hibernate會則需要執行N+1次，可想而知當資料量變大時，勢必會帶來效能問題。

DEMO

spring data jpa 、junit 5、h2database、mySql、testContainer

首先開啟相關的LOG來觀察

# 印出SQL

spring.jpa.show-sql=true

# 格式化SQL

spring.jpa.properties.hibernate.format_sql=true

logging.level.org.hibernate.SQL=DEBUG

# 印出統計數據

spring.jpa.properties.hibernate.generate_statistics=true

Company.java 公司entity

@Entity

@Table

@Data

public class Company {

@Id

@GeneratedValue(strategy = GenerationType.IDENTITY)

private Long id;

private String companyName;



@OneToMany(fetch = FetchType.LAZY, cascade = CascadeType.ALL)

@JoinColumn

private List<Employee> employees;

}

Employee.java 員工entity

@Entity

@Table

@Data

public class Employee {

@Id

@GeneratedValue(strategy = GenerationType.IDENTITY)

private Long id;

private String name;

}

CompanyRepository.java 繼承 JpaRepository

public interface CompanyRepository extends JpaRepository<Company, Long> {

}

關聯: 一間公司對多的員工 CompanyRepositoryTest 測試情境

saveTest 會儲存5間公司(Company)，每間公司各有一名員工(員工)，透過cascade all去同時儲存進DB，因為要讓兩個Test不同Transaction， saveTest 會先將結果commit，避免所有物件仍處於 entityManager 託管之下，影響測試結果。 testSelect 會透過findAll查詢所有資料後印出資料，藉此觀察LOG出的結果

@ExtendWith(SpringExtension.class)

@DataJpaTest

@Slf4j

@Transactional

class CompanyRepositoryTest {



@Autowired

private CompanyRepository companyRepository;





@Test

@Commit

void saveTest() {



for (int i = 0; i < 5; i++) {

Company company1 = new Company();

company1.setCompanyName("公司"+ i);



List<Employee> employees = new ArrayList<>();

Employee employee = new Employee();

employee.setName("員工" + i);



employees.add(employee);

company1.setEmployees(employees);

companyRepository.save(company1);

}



}



@Test

void testSelect() {

companyRepository.findAll().forEach(company -> {

log.info(company.getCompanyName() + company.getEmployees().get(0).getName());

});

}

}

測試結果可看出Hibernate會先查詢出所有公司，但並不會馬上撈出相關的員工，而是當需要印出LOG的時候，才會一筆一筆去取得員工資料，導致查詢次數就是1次查詢所有公司，加上5次(間)公司的員工，總共6(5+1)。

2020-12-02 14:20:19.993 DEBUG 10388 --- [ Test worker] org.hibernate.SQL :

select

company0_.id as id1_0_,

company0_.company_name as company_2_0_

from

company company0_

Hibernate:

select

company0_.id as id1_0_,

company0_.company_name as company_2_0_

from

company company0_

2020-12-02 14:20:20.009 DEBUG 10388 --- [ Test worker] org.hibernate.SQL :

select

employees0_.employees_id as employee3_1_0_,

employees0_.id as id1_1_0_,

employees0_.id as id1_1_1_,

employees0_.name as name2_1_1_

from

employee employees0_

where

employees0_.employees_id=?

Hibernate:

select

employees0_.employees_id as employee3_1_0_,

employees0_.id as id1_1_0_,

employees0_.id as id1_1_1_,

employees0_.name as name2_1_1_

from

employee employees0_

where

employees0_.employees_id=?

2020-12-02 14:20:20.034 INFO 10388 --- [ Test worker] c.r.h.repository.CompanyRepositoryTest : 公司0員工0

2020-12-02 14:20:20.035 DEBUG 10388 --- [ Test worker] org.hibernate.SQL :

select

employees0_.employees_id as employee3_1_0_,

employees0_.id as id1_1_0_,

employees0_.id as id1_1_1_,

employees0_.name as name2_1_1_

from

employee employees0_

where

employees0_.employees_id=?

Hibernate:

select

employees0_.employees_id as employee3_1_0_,

employees0_.id as id1_1_0_,

employees0_.id as id1_1_1_,

employees0_.name as name2_1_1_

from

employee employees0_

where

employees0_.employees_id=?

2020-12-02 14:20:20.037 INFO 10388 --- [ Test worker] c.r.h.repository.CompanyRepositoryTest : 公司1員工1

2020-12-02 14:20:20.037 DEBUG 10388 --- [ Test worker] org.hibernate.SQL :

select

employees0_.employees_id as employee3_1_0_,

employees0_.id as id1_1_0_,

employees0_.id as id1_1_1_,

employees0_.name as name2_1_1_

from

employee employees0_

where

employees0_.employees_id=?

Hibernate:

select

employees0_.employees_id as employee3_1_0_,

employees0_.id as id1_1_0_,

employees0_.id as id1_1_1_,

employees0_.name as name2_1_1_

from

employee employees0_

where

employees0_.employees_id=?

2020-12-02 14:20:20.039 INFO 10388 --- [ Test worker] c.r.h.repository.CompanyRepositoryTest : 公司2員工2

2020-12-02 14:20:20.042 DEBUG 10388 --- [ Test worker] org.hibernate.SQL :

select

employees0_.employees_id as employee3_1_0_,

employees0_.id as id1_1_0_,

employees0_.id as id1_1_1_,

employees0_.name as name2_1_1_

from

employee employees0_

where

employees0_.employees_id=?

Hibernate:

select

employees0_.employees_id as employee3_1_0_,

employees0_.id as id1_1_0_,

employees0_.id as id1_1_1_,

employees0_.name as name2_1_1_

from

employee employees0_

where

employees0_.employees_id=?

2020-12-02 14:20:20.050 INFO 10388 --- [ Test worker] c.r.h.repository.CompanyRepositoryTest : 公司3員工3

2020-12-02 14:20:20.051 DEBUG 10388 --- [ Test worker] org.hibernate.SQL :

select

employees0_.employees_id as employee3_1_0_,

employees0_.id as id1_1_0_,

employees0_.id as id1_1_1_,

employees0_.name as name2_1_1_

from

employee employees0_

where

employees0_.employees_id=?

Hibernate:

select

employees0_.employees_id as employee3_1_0_,

employees0_.id as id1_1_0_,

employees0_.id as id1_1_1_,

employees0_.name as name2_1_1_

from

employee employees0_

where

employees0_.employees_id=?

2020-12-02 14:20:20.053 INFO 10388 --- [ Test worker] c.r.h.repository.CompanyRepositoryTest : 公司4員工4

2020-12-02 14:20:20.060 INFO 10388 --- [ Test worker] i.StatisticalLoggingSessionEventListener : Session Metrics {

611700 nanoseconds spent acquiring 1 JDBC connections;

0 nanoseconds spent releasing 0 JDBC connections;

975200 nanoseconds spent preparing 6 JDBC statements;

1671200 nanoseconds spent executing 6 JDBC statements;

0 nanoseconds spent executing 0 JDBC batches;

0 nanoseconds spent performing 0 L2C puts;

0 nanoseconds spent performing 0 L2C hits;

0 nanoseconds spent performing 0 L2C misses;

0 nanoseconds spent executing 0 flushes (flushing a total of 0 entities and 0 collections);

34500 nanoseconds spent executing 1 partial-flushes (flushing a total of 0 entities and 0 collections)

}

如何解決N+1問題

這邊先列出比較簡單的作法，其實使用Criteria query也能解決這個問題，EntityGraph也有更複雜的情境可以使用，但就不繼續深究了

JPQL的方式，使用 JOIN FETCH EntityGraph 是JPA2.1 之後提供的一種solution，可以比較動態的選擇如何讀取資料。

@Query("SELECT p FROM Company p LEFT JOIN FETCH p.employees")

List<Company> findWithoutNPlusOne();



@EntityGraph(attributePaths = {"employees"})

List<Company> findAll();

調整後LOG可以看出僅執行一次SQL而已

2020-12-02 15:48:52.746 DEBUG 17908 --- [ Test worker] org.hibernate.SQL :

select

company0_.id as id1_0_0_,

employees1_.id as id1_1_1_,

company0_.company_name as company_2_0_0_,

employees1_.name as name2_1_1_,

employees1_.employees_id as employee3_1_0__,

employees1_.id as id1_1_0__

from

company company0_

left outer join

employee employees1_

on company0_.id=employees1_.employees_id

Hibernate:

select

company0_.id as id1_0_0_,

employees1_.id as id1_1_1_,

company0_.company_name as company_2_0_0_,

employees1_.name as name2_1_1_,

employees1_.employees_id as employee3_1_0__,

employees1_.id as id1_1_0__

from

company company0_

left outer join

employee employees1_

on company0_.id=employees1_.employees_id

2020-12-02 15:48:52.787 INFO 17908 --- [ Test worker] c.r.h.repository.CompanyRepositoryTest : 公司0員工0

2020-12-02 15:48:52.788 INFO 17908 --- [ Test worker] c.r.h.repository.CompanyRepositoryTest : 公司1員工1

2020-12-02 15:48:52.788 INFO 17908 --- [ Test worker] c.r.h.repository.CompanyRepositoryTest : 公司2員工2

2020-12-02 15:48:52.788 INFO 17908 --- [ Test worker] c.r.h.repository.CompanyRepositoryTest : 公司3員工3

2020-12-02 15:48:52.788 INFO 17908 --- [ Test worker] c.r.h.repository.CompanyRepositoryTest : 公司4員工4

2020-12-02 15:48:52.793 INFO 17908 --- [ Test worker] i.StatisticalLoggingSessionEventListener : Session Metrics {

689600 nanoseconds spent acquiring 1 JDBC connections;

0 nanoseconds spent releasing 0 JDBC connections;

1469200 nanoseconds spent preparing 1 JDBC statements;

448800 nanoseconds spent executing 1 JDBC statements;

0 nanoseconds spent executing 0 JDBC batches;

0 nanoseconds spent performing 0 L2C puts;

0 nanoseconds spent performing 0 L2C hits;

0 nanoseconds spent performing 0 L2C misses;

0 nanoseconds spent executing 0 flushes (flushing a total of 0 entities and 0 collections);

29800 nanoseconds spent executing 1 partial-flushes (flushing a total of 0 entities and 0 collections)

}

Batch update/insert

當我們有大批資料需要insert Or Update的時候，常常需要考慮到效能問題，避免前端TimeOut(暫不考慮 Event-driven)，至少也要盡量縮短使用者的等待時間，這時開發速度很快的JPA就會產生一些效能問題。

DEMO

最簡單也最基本要做到就是不應該將 save 放到迴圈當中，增加DB的負擔

private final int totalCnt = 1000;



@Test

void saveEachTimeTest() {



for (int i = 0; i < totalCnt; i++) {

School sc = new School();

sc.setId(i+totalCnt);

sc.setSchoolName("學校"+ i);



schoolRepository.save(sc);

}



}

@Test

void saveAllTest() {

List<School> schools = new ArrayList<>();

for (int i = 0; i < totalCnt; i++) {

School company1 = new School();

company1.setSchoolName("學校"+ i);

company1.setId(i);

schools.add(company1);

}

schoolRepository.saveAll(schools);

}

一千筆大約就是10秒跟2秒的差異，而且仔細查看LOG其實並沒有真的使用到JDBC batches

2020-12-02 17:31:10.384 INFO 16916 --- [ Test worker] i.StatisticalLoggingSessionEventListener : Session Metrics {

1810300 nanoseconds spent acquiring 1 JDBC connections;

0 nanoseconds spent releasing 0 JDBC connections;

46370600 nanoseconds spent preparing 1000 JDBC statements;

2126224300 nanoseconds spent executing 1000 JDBC statements;

0 nanoseconds spent executing 0 JDBC batches;

0 nanoseconds spent performing 0 L2C puts;

0 nanoseconds spent performing 0 L2C hits;

0 nanoseconds spent performing 0 L2C misses;

2419325600 nanoseconds spent executing 1 flushes (flushing a total of 1000 entities and 0 collections);

0 nanoseconds spent executing 0 partial-flushes (flushing a total of 0 entities and 0 collections)

}

如何開啟

application.properties 增加相關設定，設定說明

spring.jpa.properties.hibernate.jdbc.batch_size=50

spring.jpa.properties.hibernate.jdbc.batch_versioned_data=true

spring.jpa.properties.hibernate.jdbc.order_inserts=true

spring.jpa.properties.hibernate.jdbc.order_updates=true

# spring.datasource.url= xxx?reWriteBatchedInserts=true <= url需補上設定

調整後可以看到的確實際上是有使用了 JDBC batches

2020-12-02 18:23:33.045 INFO 8864 --- [ Test worker] i.StatisticalLoggingSessionEventListener : Session Metrics {

2280700 nanoseconds spent acquiring 1 JDBC connections;

0 nanoseconds spent releasing 0 JDBC connections;

51895200 nanoseconds spent preparing 1001 JDBC statements;

2176671600 nanoseconds spent executing 1000 JDBC statements;

145272700 nanoseconds spent executing 20 JDBC batches;

0 nanoseconds spent performing 0 L2C puts;

0 nanoseconds spent performing 0 L2C hits;

0 nanoseconds spent performing 0 L2C misses;

218915000 nanoseconds spent executing 1 flushes (flushing a total of 1000 entities and 0 collections);

0 nanoseconds spent executing 0 partial-flushes (flushing a total of 0 entities and 0 collections)

}

但效能上似乎沒有差多少，原因是如果不為isNew的情況，JPA會走merge，就會先select出來在進行updateOrSave，導致效能低下。 SimpleJpaRepository.java

@Transactional

@Override

public <S extends T> S save(S entity) {



Assert.notNull(entity, "Entity must not be null.");



if (entityInformation.isNew(entity)) {

em.persist(entity);

return entity;

} else {

return em.merge(entity);

}

}

此時如果能夠確定此操作永遠只會是insert而不會有update，可以在entity實作 import org.springframework.data.domain.Persistable ， 將isNew 調整為 true，就不會先去select，實測一千筆從3秒左右降為0.5秒，這應該是JPA所能達到的效能極限，若有其他方式希望可以留言告知。

@Entity

@Table

@Data

public class School implements Persistable<Integer>{

@Id

private Integer id;

private String schoolName;



@Override

public boolean isNew() {

return true;

}

}

BATCH UPDATE

如果真的是需要UPDATE，建議不要使用jpa，改用JDBC TEMPLATE來實作BATCH UPDATE，實測約O.2秒，其實是最快的

@Repository

public class BatchSchoolRepository {



@Autowired

private JdbcTemplate jdbcTemplate;



public int[] insertOrUpdateSchool(List<School> schools) {



StringBuilder sql = new StringBuilder();

sql.append("INSERT INTO school");

sql.append(

" (id, school_name) ");

sql.append(

"VALUES(:id, :school_name) ");

sql.append("ON DUPLICATE KEY UPDATE");

sql.append(" id = VALUES(id),");

sql.append(" school_name = VALUES(school_name)");



List<Map<String, Object>> batchValues = new ArrayList<>(schools.size());

for (School school : schools) {

batchValues.add(

new MapSqlParameterSource("id", school.getId())

.addValue("school_name", school.getSchoolName())

.getValues());

}



SqlParameterSource[] batch = SqlParameterSourceUtils.createBatch(batchValues.toArray());

NamedParameterJdbcTemplate namedParameterJdbcTemplate =

new NamedParameterJdbcTemplate(jdbcTemplate);

return namedParameterJdbcTemplate.batchUpdate(sql.toString(), batch);

}

}

注意事項

若使用MySql DB，ID不可以使用generator自動產生，否則預設會關閉batch update/insert 若使用Mysql DB，URL需補上rewriteBatchedStatements=true才會生效 跑test時 要掛 @SpringBootTest(webEnvironment = RANDOM_PORT) 不能只有 @DataJpaTest ，推測是沒有吃到相關測定，還需要深入研究。

總結

JPA 雖然非常好用，增加開發效率，但在使用的同時要注意，若專案範圍較大，資料量不低的情況，建議同時有兩種repository，大量的情境改使用jdbc template，單筆使用JPA，就可以同時享受JPA的開發速度，也不用犧牲效能。

參考資料

https://medium.com/sipios/eliminate-spring-hibernate-n-plus-1-queries-f0bcf6a83de2 https://persistencelayer.wixsite.com/springboot-hibernate/post/the-best-way-to-batch-inserts-via-saveall-iterable-s-entities