open-uri, Easy-to-Use andExtensible Virtual File System
Tanaka Akira
akr@m17n.org
Free Software Initiative Group,
Information Technology Research Institute,
National Institute of Advanced Industrial Science and Technology (AIST)

2005–10–14
open-uri, Easy-to-Use and Extensible Virtual File System – p. 1

Table of Contents
Who am I?
• How to use open-uri••

Why open-uri?

open-uri and net/http• How to design easy-to-use API• Easy-to-use v.s. security••

VFS – Virtual File Systemopen-uri, Easy-to-Use and Extensible Virtual File System – p. 2

Who am I?

open-uri, Easy-to-Use and Extensible Virtual File System – p. 3

Who am I (1)The author of open-uri andseveral standard libraries:
open-uri.rb, pathname.rb, time.rb, pp.rb,
prettyprint.rb, resolv.rb, resolv-replace.rb, tsort.rb

open-uri, Easy-to-Use and Extensible Virtual File System – p. 4

Who am I (2)Contribution for various
classes and methods.

IO without stdio
IO#read and readpartial
TimeTime.utc

Time#utc_offset

allocate marshal_dump marshal_load
Regexp#to_s

Regexp.union

Process.daemon
fork kills all other threads

open-uri, Easy-to-Use and Extensible Virtual File System – p. 5

Who am I (3)I reports many bugs, over 100/year.
•core dump

test failure• build problem• mismatch between doc. and imp.
• etc.
•

open-uri, Easy-to-Use and Extensible Virtual File System – p. 6

Who am I (4)I wrote several non-standard libraries.
htree• webapp• amarshal• ruby-tzfile••

vfs-simple

open-uri, Easy-to-Use and Extensible Virtual File System – p. 7

How to Use
open-uriopen-uri, Easy-to-Use and Extensible Virtual File System – p. 8

Simple Usage
require ’open-uri’open("http://www.ruby-lang.org"){|f|
print f.read}

Similar to open files.
open-uri, Easy-to-Use and Extensible Virtual File System – p. 9

Whyopen-uri?
open-uri, Easy-to-Use and Extensible Virtual File System – p. 10

Why open-uri?
••

Easy-to-Use API
VFS: Not Only HTTP

open-uri, Easy-to-Use and Extensible Virtual File System – p. 11

open-uriandnet/http
open-uri, Easy-to-Use and Extensible Virtual File System – p. 12

net/http has Too Many Ways
Net::HTTP.get_print
Net::HTTP.get
Net::HTTP.start{|h| h.get }Net::HTTP.start{|h|h.request_get{|r|}}h = Net::HTTP.new; h.start {|h|h.request_get {|r| } }h = Net::HTTP.new; h.start {|h|q = HTTP::Get.newh.request(q){|r|}}open-uri, Easy-to-Use and Extensible Virtual File System – p. 13

open-uri has Fewer Ways
open(uri) {|f| }uri.open {|f| }uri.read•

Save User’s Memory•

Reuse User’s Knowledge
open-uri, Easy-to-Use and Extensible Virtual File System – p. 14net/http: get and print
Net::HTTP.get_print(URI("http://host"))print Net::HTTP.get(
URI("http://host"))open-uri, Easy-to-Use and Extensible Virtual File System – p. 15

open-uri: get and print
open("http://host") {|f|print f.read}print URI("http://host").read

open-uri, Easy-to-Use and Extensible Virtual File System – p. 16

get and print
net/http:• Net::HTTP.get_print : print only
• Net::HTTP.get : good
open-uri:• open : conventional• URI().read : short, polymorphism

open-uri, Easy-to-Use and Extensible Virtual File System – p. 17

Why Easy?
open("http://host")•No new construct•

Users don’t need to learn.

open-uri respects userknowledges.
open-uri, Easy-to-Use and Extensible Virtual File System – p. 18

net/http: headers
Net::HTTP.start ( "host" ) {|h|r = h. get ( "/" ,"User-Agent" => "bar")p r["Content-Type"]
print r. body}• No URI anymore• No Net::HTTP.get anymore• Net::HTTP.start, Net::HTTP#get andNet::HTTP::Response#body instead.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 19

open-uri: headersopen("http://host","User-Agent" => "bar") {|f|p f.content_type
print f.read}• Still URI• Still open method• Fewer things to learn.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 20

net/http: SSL
require "net/https"
h = Net::HTTP.new("host", 443 )h.use_ssl = true
h.ca_file = "/etc/ssl/certs/ca-certificates.crt"
h.verify_mode = OpenSSL::SSL::VERIFY_PEER
h. start {r = h.get("/")
print r.body}• Different library: net/https• Net::HTTP.new and Net::HTTP#start• Different port
• Server verification not by default

open-uri, Easy-to-Use and Extensible Virtual File System – p. 21

open-uri: SSL
open("https://host") {|f|print f.read}• Still URI• Still open method• Server verification by default• No new library.
• No new methods. Fewer things to learn.
open-uri, Easy-to-Use and Extensible Virtual File System – p. 22

net/http: proxyklass = Net::HTTP::Proxy("proxy", 8080)
klass.start("host") {|h|r = h.get("/")
print r.body}• New method: Net::HTTP::Proxy

open-uri, Easy-to-Use and Extensible Virtual File System – p. 23

open-uri: proxy% http_proxy=http://proxy:8080/% export http_proxy• Conventional environment variable supported• No new methods. An user might know thisalready.
• Fewer things to learn.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 24

net/http: basic authNet::HTTP.start("host") {|h|q = Net::HTTP::Get .new("/")
q.basic_auth "user", "pass"r = h. request (q)print r.body}• New class: Net::HTTP::Get• New method: Net::HTTP#request

open-uri, Easy-to-Use and Extensible Virtual File System – p. 25

open-uri: basic authopen("http://host",:http_basic_authentication=>["user", "pass"]) {|f|print f.read}• Still URI• Still open method• New option: :http_basic_authentication• No new methods. Fewer things to learn.
open-uri, Easy-to-Use and Extensible Virtual File System – p. 26

How toDesignEasy-to-Use
API
open-uri, Easy-to-Use and Extensible Virtual File System – p. 27

How to Design Easy-to-UseAPI••

Save brain power
Evolve gradually

open-uri, Easy-to-Use and Extensible Virtual File System – p. 28

Save Brain Power

Fewer Things to Learn• Fewer constructs for pragmatic usages• Huffman coding• DRY• No configuration is good configuration• Reuse user knowledge• Infrastructure friendly

open-uri, Easy-to-Use and Extensible Virtual File System – p. 29

Fewer Constructs for Pragmatic Usages
Fewer constructs decrease things tolearn
open v.s. Net::HTTP.get,Net::HTTP#get, etc.
• This is not minimalism.
• The target of "fewer" is not allconstructs.
•Pragmatic usages should be supportedby small constructs.
open-uri, Easy-to-Use and Extensible Virtual File System – p. 30

Fewer Constructs (2)frequently used for
rarely used pragmatic usages
convenience
primitives
methods

should be smaller
open-uri, Easy-to-Use and Extensible Virtual File System – p. 31

Ex. net/http and open-uri
Methods frequently used:
net/http Net::HTTP.start, Net::HTTP#get
open-uri open
open-uri’s fewer constructs supports
much more features.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 32

Huffman Coding• Shorter for frequent things• Longer for rare things

Optimize for frequent things.
Ex: p

open-uri, Easy-to-Use and Extensible Virtual File System – p. 33

Huffman Coding (2)rarely used frequently usedconvenience
primitives
methods
longer methods

shorter methods
open-uri, Easy-to-Use and Extensible Virtual File System – p. 34

Ex. pp obj

Very frequently used
• Bad name in common sense• Almost no problem because everyone
knows•open-uri, Easy-to-Use and Extensible Virtual File System – p. 35

Ex. pp and y
Bad name in common sense• Problematic than p because noteveryone knows•

open-uri, Easy-to-Use and Extensible Virtual File System – p. 36

Ex. to_s and to_str
to s shorter. frequently used.
to str longer. internal use.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 37

Ex. def
def shorter. frequently used.
define method longer. not encouraged.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 38

Ex. time.rb
Time.parse frequently used.
Time.strptime generic. needs to learn theformat.

Time.parse is less flexible but enough formost cases, and easy to learn.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 39

Candidates for Huffman Coding
Method name• Other name• Convenience method• Language syntax• etc.
•

open-uri, Easy-to-Use and Extensible Virtual File System – p. 40

Length for Huffman Coding
Number of characters• Number of nodes in AST• Editor keystrokes• etc.
•

open-uri, Easy-to-Use and Extensible Virtual File System – p. 41

Encourage Good Style••

Programmers like short code
Short code should be designed as
good style

open-uri, Easy-to-Use and Extensible Virtual File System – p. 42

DRYDon’tRepeat
Yourself
open-uri, Easy-to-Use and Extensible Virtual File System – p. 43

DRY Violation
Net::HTTP .start("host") {|h|q = Net::HTTP ::Get.new("/")
q.basic auth "user", "pass"r = h.request(q)print r.body}open-uri, Easy-to-Use and Extensible Virtual File System – p. 44

No Configuration is Good
Configuration

Things should be work well out-of-box.
SSL CA certificates• http_proxy environment variable•

open-uri, Easy-to-Use and Extensible Virtual File System – p. 45

Bad Examples
ext/iconv/config.charset• soap_use_proxy••require "irb/completion"•

RUBYOPT=rubygems

open-uri, Easy-to-Use and Extensible Virtual File System – p. 46

Reuse User Knowledge
open-uri reuse user knowledge.
• open is used to access an externalresource• If a block is given for open, it is calledwith a file object
Various knowledge about open isreused.
Fewer things to learn.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 47

Reusable Knowledge•

Ruby builtin (popular) method

Consistency• Unix• Standards: POSIX, RFC, etc.
• Metaphor•

open-uri, Easy-to-Use and Extensible Virtual File System – p. 48

Consistencybang methods• each_with_index• etc.
•Consistency violation:• Time#utc is destructive

open-uri, Easy-to-Use and Extensible Virtual File System – p. 49

Metaphor••

HTTP is a kind of a network filesystemopen-uri doesn’t support beyond filesystem: POST, etc

open-uri, Easy-to-Use and Extensible Virtual File System – p. 50

Infrastructure Friendlyemacs, vi• line oriented tools• shell and file system• web browser•

Prefer"It is easy using the legacy tool XXX"
over"It is easy using the new tool YYY"

open-uri, Easy-to-Use and Extensible Virtual File System – p. 51

Evolve Gradually•

Adaptive Huffman coding•

How to find bad API••

How to avoid
incompatibility
Incompatible change
open-uri, Easy-to-Use and Extensible Virtual File System – p. 52

Adaptive Huffman Coding
What methods are used frequently?
Long method name at first• Alias to short name later• Define convenience methods foridioms•open-uri, Easy-to-Use and Extensible Virtual File System – p. 53

Adaptive Huffman Coding (2)••Short names and operators should beused carefully
Use a long name if hesitate

Alias is not a bad thing (TMTOWTDI)• Primitives should have long names••

Define new method for idiom

open-uri, Easy-to-Use and Extensible Virtual File System – p. 54

Operators
CGI#[] and CGI#params
CGI#[] was defined unsuitably.
• Hash#[]primitive: Hash#fetch•

open-uri, Easy-to-Use and Extensible Virtual File System – p. 55

How to Find Bad API•

Repeated surprise

Often cannot remember• Idiom•

open-uri, Easy-to-Use and Extensible Virtual File System – p. 56

Repeated Surprise
Example:• Time#utc is destructive• Iconv.iconv returns an array
String#gsub(/\\/, ’\\\\’) has noeffect• etc.
Violate POLS•

open-uri, Easy-to-Use and Extensible Virtual File System – p. 57

Often Cannot Remember
Manual is required again and again forsame issue.
RubyUnit• optparse•

open-uri, Easy-to-Use and Extensible Virtual File System – p. 58

RubyUnitrequire ’runit /testcase ’require ’runit/cui/testrunner’class TestC < RUNIT ::TestCase
def test unit...
endendRUNIT::CUI::TestRunner.run(TestC.suite)
open-uri, Easy-to-Use and Extensible Virtual File System – p. 59

Test::Unit
require ’test / unit’class TestC <Test::Unit ::TestCase
def test unit
...
end
end
Test::Unit removed code for runner.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 60

optparse
require ’optparse’
ARGV.options {|q|q.on ("-h") { puts q }q.on("-v") { $VERBOSE = true }q.parse!
}open-uri, Easy-to-Use and Extensible Virtual File System – p. 61

IdiomRepeated code
• Violate DRY• An idiom may be good
• An idiom may be bad•

Bad idiom example:•

Iconv.iconv()[0]open-uri, Easy-to-Use and Extensible Virtual File System – p. 62

How to Avoid Incompatibility
Extension without Incompatibility:• new method• new keyword argument• new constants
Introducing new names has nocompatibility problem. (in most case)open-uri, Easy-to-Use and Extensible Virtual File System – p. 63

Incompatible Change

Incompatible Change
is a Bad Thing
But fixing bad API involves incompatible change,sometimes.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 64

Incompatible Change
API Migration Example•••

net/http: API version
cgi: special implementation for atransition period
fork: warning after change

IO#read: warning before change• etc.
•

open-uri, Easy-to-Use and Extensible Virtual File System – p. 65

net/http: API version
Net::HTTP has two APIs.
• Ruby 1.6: API version 1.1• Ruby 1.7: API version 1.2
API version can be switched dynamically.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 66

net/http: switch API version
Net::HTTP.version_1_1... use 1.1 API ...
Net::HTTP.version_1_2... use 1.2 API ...
•It tends to forget restore API version•

Global switch – not thread safe

open-uri, Easy-to-Use and Extensible Virtual File System – p. 67

cgi: special implementation
for a transition period
CGI#[] returns:• Ruby 1.6: an array of parameters• Ruby 1.8: transition period•

future?: a first parameter or nil

open-uri, Easy-to-Use and Extensible Virtual File System – p. 68

cgi: special implementation
for a transition period

CGI#[] returns something tweaked on
Ruby 1.8.
Try to work as both Array and String.
•••

Ruby 1.8.0: subclass of String
Ruby 1.8.1: subclass ofDelegateClass(String)
Ruby 1.8.2: extended String

open-uri, Easy-to-Use and Extensible Virtual File System – p. 69

fork: warning after change
Does fork kill other threads in childprocess?
• Ruby 1.6: No• Ruby 1.8: Yes

open-uri, Easy-to-Use and Extensible Virtual File System – p. 70

fork: warning after change% ruby -e ’Thread.new{sleep};fork’

Ruby 1.6: No warning• Ruby 1.8.0: No warning• Ruby 1.8.1: warning:fork terminates thread• Ruby 1.8.2: No warning•

open-uri, Easy-to-Use and Extensible Virtual File System – p. 71

IO#read:changewarning

before

IO#read will block even if
O_NONBLOCK is set.
• Ruby 1.8: doesn’t block• Ruby 1.9: blockopen-uri, Easy-to-Use and Extensible Virtual File System – p. 72

IO#read:changewarningbefore

IO#read will block even if
O_NONBLOCK is set.
• Ruby 1.8.2: No warning• Ruby 1.8.3: warning:
nonblocking IO#read is obsolete;
use IO#readpartial or IO#sysread• Ruby 1.9: No warning
warning only if verbose mode.
open-uri, Easy-to-Use and Extensible Virtual File System – p. 73

Easy-to-Usev.s. Security
open-uri, Easy-to-Use and Extensible Virtual File System – p. 74

Easy-to-Use v.s. Security
HTTP_PROXY• http://user:pass@host/• redirection and taint• File.open(uri)•open-uri, Easy-to-Use and Extensible Virtual File System – p. 75

VFS
Virtual File
Systemopen-uri, Easy-to-Use and Extensible Virtual File System – p. 76

VFS•

Why VFS?

What is VFS• VFS and polymorphism• Polymorphic open•

Other Resources• Other Operations••

Security Considerations

open-uri, Easy-to-Use and Extensible Virtual File System – p. 77

Why VFS?
Typical simple program:• Load an external resource• Process the resource• Store the result
VFS ease the first step.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 78

What is VFS
VFS provides:•open a http/ftp/... resource

read a http/ftp/... resource• etc.
filesystem like operations for
non-filesystem target
Polymorphism of filesystem•open-uri, Easy-to-Use and Extensible Virtual File System – p. 79

VFS and polymorphism
The polymorphism can be implemented
by:• usual method dispatch mechanism• own mechanism
open-uri uses the method dispatch forthe polymorphism.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 80

Polymorphic open
If open-uri is in effect:•open("http://...") calls
URI("http://...").open

open("ftp://...") calls
URI("ftp://...").open• etc.
•Any URI can be opened if the URI has
open method.
open-uri, Easy-to-Use and Extensible Virtual File System – p. 81

Other Resources
LDAP:
class URI::LDAP
def open(*args)...
endendopen("ldap://...") { ... }open-uri, Easy-to-Use and Extensible Virtual File System – p. 82

Other Operations••

URI("http://...").read
Other operations should be definedfor polymorphic to Pathname in future.

open-uri, Easy-to-Use and Extensible Virtual File System – p. 83

Security Considerations•

open("|...")•File.open is not affected

open-uri, Easy-to-Use and Extensible Virtual File System – p. 84

Summary••

How to design Easy-to-Use API• Save brain power• Evolve gradually
VFS by open-uri

open-uri, Easy-to-Use and Extensible Virtual File System – p. 85