Mining Resourcesin the Internet Space
Katsuro Inoue
Osaka Univeristy

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

1Mining OSS DataAnalyst

Powerful WSHugeRepository
OSS Data

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University2

Maintaining Huge Repository• Keeping the repository updated isimportant– Time consuming effort– Automatic crawling is not easy

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

3

Mining OSS DataAnalyst

Powerful AnalyzerHugeRepository
OSS Data

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

4

Mining OSS DataAnalyst

Powerful Analyzer

OSS Data
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

5

Mining OSS Data
Internet SpaceAnalyst

Powerful Analyzer

Internet Search Engines

OSS Data

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

6

Mining OSS Data
Internet Space

AnalystSmall Interfaceand Filter

Internet Search Engines

OSS Data

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

7

Code History Tracking Modeland Ichi Tracker

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

8

Developer’s Concerns
Existing Project• Origin- Who?
- When?

Code Fragment

Reusable?

- License?
- Copyright?

New Project• Evolution- Maintenance?
Developer- Popularity?
- Newer version?
…

To ease concerns, a support system is needed
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

9

Code History Tracking System?

Code Fragment
in Question
Copy SourceProj. C

Descendants

License: X
Copyright: Y

Proj. B

Proj. F

ModifyProj. E
Proj. A

Proj. G
Ancestor

ModifyCopy

Proj. D
License: X /Copyright: Y

Proj. HTime

License: X’
Copyright Y’

OSS Repositories
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

10Design Policy of Ichi Tracker
OSS repository• Target many OSS projects from old to new ones
• No crawling, no maintenance Do not have local repository,
but use external code search engines

Output quality• Find not only exactly same code fragments, but alsosimilar ones
• Lower false positive results• No real-time response Use code clone filtering to improve the output quality
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

11Code History Tracking Model
Output Results R

Input Query Q
CodeqcFragment
Code
Attributes(Optional)

Integrated CodeHistory Tracker

Ichi Tracker

attribute 1: ...
attribute 2: ...
...

SearchResults SRSearch
Query SQ

CodeFragment

CodeClones
attribute 1: ...
attribute 2: ...
...

Code
Attributesattribute 1: ...
attribute 2: ...
...

Code Search Engines
SPARS/RGoogle
CodeSearch

KodersInternet
attribute 1: ...
attribute 2: ...
...

Open Source Repositories
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

12Process of Ichi Tracker
Input
Query Q

1) WordExtractionno4) ResultsOK?

yes5) Download/
Search Result
Analysis2) Keyword
Selection6) CodeClone
Filtering3) Query
Generation

7) ResultForming

Keywords

SearchQuery SQ

OutputResults R

Header List/
FilesSearch
Results SR

Control andData Flow

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

13Evolution Pattern of Texture.java
14756 8925

jMonkeyEngine R3800

C0.9

Cover Ratio

0.8

A
0.7
1

10

2
3R3448B

0.6

R4099 12 131415

16 17 19 20 21 2223 24

26

18

0.5
110.4
2006/10/10

2007/04/28

2007/11/14

2008/06/01

2008/12/18

2009/07/06

2010/01/22

2010/08/10

1 Jmonkeyengine r3448(K)

2 Simplexe(G)

3 Simplexe(K)

4 Jmonkeyengine r3800(G)

5 The-project08(G)

6 Tank Combat Game(K)7 wrathofthetaboos(G)

8 wrathofthetaboos(K)

9 Lasthaven(G)

10 Jme-cotk(G)

11 Ardor3D(G)

12 Jmonkeyengine r4099(G)

13 Fairytale-soulfire(G)

14 Jmonkeyengine r4490(G)

15 Jmonkeyengine r4490(S)

16 Xenogeddon(G)

17 Deathsquadrendezvous(G)

18 Partiendolapana(G)

19 Tholos(G)

20 Ardor3D(G)

21 Cosmic-engine(G)

22 Fregatclient3d(G)

23 Footballmanagerdesia(G)

24 Multiplicity(G)

25 Jmerefactoring(G)

26 Java3dfh(G)

2011/02/26

Last Modify Time
: File in New BSD License: Evolution of jMonkeyEngine Project: File in zib-libpng License: Cluster of Same or Similar Files

14

Evolution Pattern of kern_malloc
1112 34Lites 1.0(G)Kernel Source
Archive - CMU Mach
3.0(K)

1965
1324

1558

36262

110.9

3

Lites 1.1.u3

4

Lites 1.1-950808(G)
The Rio (RAM I/O)
Project(K)ftp in The Universityof Edinburgh(G)575 61069270.8
837-46

Cover Ratio

26
1214
1625
18

34
28-33

47
48

20-23

61

59

49
50

35

170.766

52 53

56
51

55

6367

57

Mip-summer98

8

freeBSD/SPARC

ftp in Stockholm
9-12
University
freeBSD-cam2.1.5R

14-15 SonicOSX

540.6

71360

64

16

Labyrinth
BSD(labyrinthos)

17

Oskit

18

Psumip19

Mach0.4
1993/01/31

Unofficial OSKit
23
sourceUnofficial OSKit
24-26
source(oskit)
ftp in Stockholm
27
University

1995/10/28

1998/07/24

2001/04/19

2004/01/14

Last modified time

2006/10/10

2009/07/06

(G)
(G)
(G)
(G)
(K)

(K)
(G)

(G)

20-22 Savannah

0.5

(G)

(G)
(G)
(K)
(K)
(G)

28-33 Kame(G)

34-36 SimOS(K)
27-46 Kame
47

Netnice

48

Kame

(G)
(G)
(G)

49-50 Psumip

(G)

51

Netnice

52

Reflexprotocol

53

Netnice54

NetBSD v1.105

55

OpenBSD PV Xen

56

OpenBSD v1.73

57

Pmon

63

OpenBSD v1.74

64

Pmon

65

77466

Chord-ns3

(G)
(G)
(G)
(K)
(G)

(K)

(G)
Proyecto A.T.L.D.
58-62 GNU/hurd(extremelinux)
(K)

67

(K)
(G)
(G)

(G)
Openbsd-loongsonvc(G)

Results by G(Google Code Search)and 2012/04/01
K(Koders): File in New BSD License: File in Original BSD License

15

Oldest Files Found for Each Query File

License:SSHTools(all 339 files) Copyright:

Last Modified Time:GPL 2
2002-2003 Lee David Painter and Contributors
2007/6/23

We have found many cases of different projects, different licenses anddifferent copyrights-> We need to check very carefully when we reuse SSHTools code
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

16

Usefulness
With simple check of the output of Ichi Tracker, we can get
useful information for the history and evolution of code
• Origin
Reusable?

- Who?

EASE- When?
- License?

IchiTracker- Copyright?

• Evolution
Results- Maintenance?
- Popularity?

- Newer version?
…
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University17

Approach and Process• Choosing good code search engines is a key toget high quality results
GCS >> Koders > SPARS/R(GCS, Koders, SPARS/R) >> (Google, Bing)
Need good code search engine available!

• Keyword selection strategy:
Incremental strategy: try 1, 2, … keywords until theheader list becomes less than 50– Decrement strategy, random, less frequently-used keywords,comment keywords, short keywords, …

Less effective
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

18

Other Issues• Performance– Case Studies A and B: 1 to 4 min.
– Heavily depend on the code search engines andnetwork performance
Acceptable as non-interactive support system• Quality of search result– Non-removed rate at the code clone filtering is anindicator of effectiveness of keyword searche.g., 0.46 (Case Study A(1) default setting)– The final output contains no false positives results

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

19

Pros and Cons
+ No huge repository+ No maintenance cost and inconsistency+ Easy to get the initial results‒ Hard to control search results (engines) indetails‒ Fragile results depending on searchengines
Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

20

Mining Resources in the Internet Space
Can we use this analogy to othercases than the code search?

AnalystInternetSmall Interfaceand Filter

Internet Search Engines

OSS Data

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

21

Thank you!

Software Engineering Laboratory, Department of Computer Science, Graduate School of Information Science and Technology, Osaka University

22