Challenges in Mining Whole
Software Universe
Katsuro Inoue
Osaka University

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

Analyzing Evolution of kern_malloc
1112 3

4Lites 1.0(G)Kernel Source
Archive - CMU Mach
3.0(K)

1965
1324

1558

36262

110.9

3Lites 1.1.u3

4

Lites 1.1-950808(G)
The Rio (RAM I/O)
Project(K)ftp in The Universityof Edinburgh
(G)

575 6

1069270.8
837-46

Cover Ratio

26
12141625
1834
28-33

52 53
4748

20-236159

495035170.766

5651

Mip-summer98

8

freeBSD/SPARC

9-12

ftp in Stockholm
University

13

freeBSD-cam2.1.5R

14-15 SonicOSX

63

16

Labyrinth
BSD(labyrinthos)

17

Oskit

18

Psumip

19

Mach

54

67

57

0.6

760

55

64

Unofficial OSKit
sourceUnofficial OSKit
24-26
source(oskit)
ftp in Stockholm
27
University
23

0.4
1993/01/31

1995/10/28

1998/07/24

2001/04/19

2004/01/14

Last modified time

2006/10/10

2009/07/06

(G)
(G)
(G)
(G)
(K)

(K)
(G)

(G)

20-22 Savannah

0.5

(G)

(G)
(G)
(K)
(K)
(G)

28-33 Kame(G)

34-36 SimOS(K)
27-46 Kame
47

Netnice48

Kame49-50 Psumip
51

Netnice

52

Reflexprotocol

53

Netnice54

NetBSD v1.105

55

OpenBSD PV Xen

56

OpenBSD v1.73

57

Pmon

63

OpenBSD v1.74

64

Pmon

65

774

66

Chord-ns3

67

Openbsd-loongsonvc

(G)
(G)
(G)
(G)
(G)
(G)
(G)
(K)
(G)

(K)

(G)
Proyecto A.T.L.D.
58-62 GNU/hurd(extremelinux)
(K)
(K)
(G)
(G)
(G)
(G)

Results by G(Google Code Search)and 2012/04/01
K(Koders): File in New BSD License: File in Original BSD License

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University2Analyzing Reuse of Outdated Libraries
Vulnerability of 50 OSS Projects Using libpng
4320

v1.0.11v1.2.1v1.2.7v1.2.8v1.2.5v1.2.12v1.2.16v1.2.21v1.2.22v1.2.23v1.2.24v1.2.27v1.2.29v1.2.32v1.2.33
v1.2.34v1.2.35v1.2.37v1.2.39v1.2.40v1.2.42v1.4.1v1.2.43v1.4.2v1.4.4v1.2.44v1.4.6beta06
v1.5.1v1.5.4v1.5.7v1.2.46v1.4.8v1.2.49v1.5.9v1.5.10v1.5.12v1.5.13

1Vulnerabilities reported

No defects reported

Result from Google Code Search and Koders

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University3Experience and Concern
Mining source code repositories, e.g.,
SourceForge, Github, Open Hub, Google Code, Marven, ...
(BlackDuck)– Outcomes heavily depend on repository
contents– Aren't we mining a small world?
– There may be many other source code
contents in the universe
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

Whole Software Universe 𝑈• Whole Software Universe𝑈 ≡ 𝐶𝑜𝑙𝑙𝑒𝑐𝑡𝑖𝑜𝑛 𝑜𝑓 𝐴𝑙𝑙 𝑆𝑜𝑓𝑡𝑤𝑎𝑟𝑒
𝐷𝑒𝑣𝑒𝑙𝑜𝑝𝑒𝑑 𝑏𝑦 𝐻𝑢𝑚𝑎𝑛 𝑖𝑛 𝑡ℎ𝑒 𝑃𝑎𝑠𝑡
– Open source software– Personally-developed software– Proprietary software... any others• 𝑃 : Set of all meaningful software(a countable infinite set)
• 𝑈⊆𝑃
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

Questions for 𝑈?

A) How do we get 𝑈?
B) What do we minefrom 𝑈?

C) How do we mine 𝑈?
D) Why do we mine 𝑈?

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

A) How Do We Get 𝑈?
• No one knows actual 𝑈• So we would collect many repositories, andconstruct a subset 𝑈 ′ ⊆ 𝑈• 𝑈 ′ should be as large as possible, of course• 𝑈 ′ should reflect characteristics of 𝑈
• Challenges– Collecting and unifying different repositories into 𝑈 ′• Duplication, coherence, ...

– Performance and capacity for 𝑈 ′– Updating and maintaining 𝑈 ′Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

B) What Do We Mine from 𝑈?
Examples• Simple metrics of 𝑈 over history– Size 𝑈 𝑡1 , |𝑈|𝑡2 , …
– Language usage…

• Density of 𝑈 with respect to 𝑃
• History and evolution of code 𝑐 in 𝑈
– Origin version of 𝑐
– Closely related code 𝑐′ (clone, variation, family, ...)– Future prediction for 𝑐
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University′C) How Do We Mine 𝑈 (𝑈 )?
1. Direct mining– Good model– Powerful machine

2. Indirect mining– Use external services– Reconstruct mining result from those externalservices

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

Direct Mining

𝑼′
Copy of 𝑼′

𝑼
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

Indirect Mining

Want to know about 𝑼′
MashupEngine𝑼′Query Decomposition
and Result Composition

𝑼
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

D) Why Do We Mine 𝑈?
Objectives of mining 𝑈• Reuse and knowledge transfer– We do not want to reinvent the wheel• Historical Archive– Frontier's wisdom...

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

Discussion!
• Is it interesting research topics?
• Can we get useful research results?
• Is it feasible research target?

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

Thank you

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka UniversitySoftware Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University