# Python Concepts/Files
## Objective

* Learn a little bit about files.
* Learn about the built-in function open.
* Learn how to read, write to, and seek in a file.
* Learn about abstractions that act like files.
* Learn about the optional parameters for open.
* Learn how to rename, remove, move, and create files.

## Lesson

* What's A File
* With work on a computer, files are usually used in daily tasks. You may spend your days writing word processor documents for a news company or you may like to listen to your mp3 files during your free time. You most likely already have an abstract idea of what a file is: a piece of information that's stored on a disk.
* So what exactly is a file?
* A file is just a group of 1's and 0's that are stored on the disk. Since the operating system takes care of managing them, you don't have to worry about their technical details.
* The data within a file may be as simple as a few words of text, or nothing at all (/dev/null); the data may be the audio and video of one of your favorite movies;
* a file may contain enormous quantities of numbers to be used to predict the path of a hurricane or to predict the existence of
* extra-terrestrial planets, or to monitor the national debt.
* Whatever the size and shape of a file, the usual operations on files are open, read and/or write, and close.
* To open a single file in Python, use the built-in function open. This code illustrates opening,
* reading and closing a file:
* ifile = open('test.txt') # ifile (input file) is a file object
* # Same as ifile = open('test.txt', 'rt') where 'rt' (default) means 'read text'.
* for line in ifile : # ifile behaves like an iterable
* print (line, end='')
* ifile.close
* The content of file 'test.txt':
* Hello, world!
* Hola, mundo!
* Goodbye, world!
* Technical definition of a file object
* >>> import io
* >>> f = open('test1.txt') ; f
* <_io.TextIOWrapper name='test1.txt' mode='r' encoding='UTF-8'>
* >>> isinstance(f, io.TextIOWrapper)
* True
* >>> f.close
* >>> f
* <_io.TextIOWrapper name='test1.txt' mode='r' encoding='UTF-8'>
* >>> isinstance(f, io.TextIOWrapper)
* True # Still valid, even though f is closed.
* >>> f.readable
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* ValueError: I/O operation on closed file
* >>> # Open file in binary mode:
* >>> f = open('test1.txt', 'r+b') ; f
* <_io.BufferedRandom name='test1.txt'>
* >>> isinstance(f, io.BufferedRandom)
* True
* >>> f.close
* >>> f
* <_io.BufferedRandom name='test1.txt'>
* >>> isinstance(f, io.BufferedRandom)
* True # Still valid, even though f is closed.

## Opening A File

* The concept of opening a file seems simple enough, but it leads to significant questions:
* Does the file exist? If not, why not?
* If the file exists, is it where you expect it to be? If so, do you have permission to open it? You may be able to open it for reading,
* but what about writing or truncating the file?
* If you open the file, is it OK if somebody else opens it while you have it open? You might decide to lock all others
* out of the file while you have it open. If so, normal etiquette requires that you do what you have to do quickly
* and then close it so that others may access it.
* Computer scientists prepare for errors and handle them gracefully. Therefore, the above code is rewritten to handle errors:
* inputFile = 'test.txt'
* # Open the file
* status = 0
* try:
* ifile = open(inputFile)
* except:
* print ("Error detected when opening ''.".format(inputFile))
* status = 99
* if status : exit (status)
* # Read the file
* status = 0
* try:
* for line in ifile :
* print (line, end='')
* except:
* print ("Error detected when reading ''.".format(inputFile))
* status = 98
* if status : exit (status)
* # Close the file
* status = 0
* try:
* ifile.close
* except:
* print ("Error detected when closing ''.".format(inputFile))
* status = 97
* if status : exit (status)
* exit (0)
* Handling Errors
* In the example immediately above there is more error-handling code than operational code.
* If you think this is unrealistic, remember that software engineers are notorious for overestimating
* their ability and underestimating the time to complete a given project.
* Simple mistakes can lead to disastrous and expensive consequences.
* Milstar: Military Strategic and Tactical Radar
* The third launch in the series, 30 April 1999, failed because an engineer entered one parameter as -0.1992476
* instead of the correct -1.992476. More than one billion dollars (that's billion with a 'b') was
* wasted.
* "Milstar satellite overview" This page doesn't mention the failed third launch.
* "History of Milstar" From Wikipedia.
* "A single error can kill a mission"
* "Examples from the Launch World"
* Close to home
* The following code is copied directly from a famous instructional book for Perl:
* opendir(ETC, "/etc") || die "no etc?";
* foreach $name (sort readdir(ETC)) {
* print "$name\n";
* close(ETC);
* Can you spot the error? Rewrite the code to catch potential errors:
* opendir(ETC, "/etc") || die "no etc?";
* foreach $name (sort readdir(ETC)) {
* print "$name\n";
* close(ETC) || die "";
* Execution of this piece of code fails at close(ETC) || die ""; The code should be:
* opendir(ETC, "/etc") || die "no etc?";
* foreach $name (sort readdir(ETC)) {
* print "$name\n";
* closedir(ETC) || die '';
* Your attitude may be: "It doesn't matter about the closedir because the operating system closes the directory
* when the application exits." If so, the code should be:
* opendir(ETC, "/etc") || die "no etc?";
* foreach $name (sort readdir(ETC)) {
* print "$name\n";
* exit (0);
* Catching errors often reveals simple mistakes in software that can go undetected for a long time.
* python's with statement
* python's with statement simplifies handling errors during operations on files:
* with open("test.txt", "r") as file:
* print ( 'status1 =', file.closed)
* print ( 'status2 =', file.closed)
* try:
* with open("test.txt", "r") as file:
* print ( 'status3 =', file.closed)
* raise NameError
* except:
* print ('Error detected in "with" statement.')
* print ( 'status4 =', file.closed)
* On exiting the body of the with statement (for any reason) python closes an open file:
* status1 = False
* status2 = True
* status3 = False
* Error detected in "with" statement.
* status4 = True

## Information about the file

* Information available after opening
* >>> f = open ('test.txt')
* >>> f
* <_io.TextIOWrapper name='test.txt' mode='r' encoding='UTF-8'> # file object
* >>> f = open ('test.txt') # File is silently reopened without error.
* >>> f.closed
* False # File is open
* >>> f.fileno
* 3 # The underlying file descriptor, an int.
* >>> f.isatty
* False # Input is not coming from a terminal.
* >>> f.readable
* True # We expect the file to be readable. It was opened without error.
* >>> f.seekable
* True # We can cause the file to seek to any desired position within the file.
* >>> f.tell
* 0 # Internal pointer is at beginning of file. This is expected after opening file.
* >>> f.writable
* False # Not writable, opened for reading only. An attempt to write or truncate raises OSError.
* >>> # Size of file:
* >>> f.seek(0,2) # Change stream position to 0 bytes from end of file.
* 43 # At end of file. Therefore, file contains 43 bytes.
* >>> f.tell
* 43 # Current position of stream.
* Function os.stat
* Function os.stat(path, *, dir_fd=None, follow_symlinks=True) provides more information about the file. It returns
* a stat_result object.
* >>> import os
* >>> info = os.stat('test1.txt')
* >>> isinstance(info, os.stat_result)
* True
* >>> info
* os.stat_result(st_mode=33188, st_ino=24051862, st_dev=16777218, st_nlink=1, st_uid=501, st_gid=20, st_size=249, st_atime=1505769137, st_mtime=1505476400, st_ctime=1505476401)
* >>> info[1]
* 24051862
* >>> info.st_ino
* 24051862
* >>> info[6]
* 249
* >>> info.st_size
* 249
* >>> # The following additional info is platform dependent:
* >>> info.st_blocks
* 8
* >>> info.st_blksize
* 4096
* >>> info.st_birthtime
* 1505476400.0
* On the Unix command line:
* $ ls -laid test1.txt
* 24051862 -rw-r--r-- 1 user staff 249 Sep 15 06:53 test1.txt
* $ date -r 1505476401
* Fri Sep 15 06:53:21 CDT 2017
* info.st_ino is the inode displayed by the Unix command ls.
* info.st_size is the size displayed by the Unix command ls.
* info.st_ctime is the creation time displayed by the Unix command ls.

## Seeking within a text file

* Within text files the method f.seek has limited functionality:
* >>> f = open ('test.txt', mode='rt')
* >>> f.seek( 16, 0 ) # Put beginning of stream at position 16 relative to beginning of file.
* 16
* >>> f.seek( 13 ) # Same as f.seek( 13, 0 )
* 13
* >>> f.seek( -5, 0 ) # Try to position stream before beginning of file.
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* ValueError: negative seek position -5
* >>> f.tell
* 13 # Position of stream pointer is unchanged.
* >>> f.seek( -5, 1 ) # Try to position stream -5 bytes relative to current position:
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* io.UnsupportedOperation: can't do nonzero cur-relative seeks
* >>> f.tell
* 13 # Position of stream pointer is unchanged.
* >>> f.seek( f.tell+4 ) # Equivalent of f.seek( 4, 1 ).
* 17
* >>> f.seek( 0,2 ) # Position stream at 0 bytes relative to end-of-file.
* 43
* >>> f.seek( -5, 2 ) # Try to position stream -5 bytes relative to end-of-file:
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* io.UnsupportedOperation: can't do nonzero end-relative seeks
* >>> f.seek( f.seek(0,2) - 7 ) # Equivalent of f.seek( -7, 2 ).
* 36
* >>> f.seek( 1234567 )
* 1234567
* >>> f.tell
* 1234567 # Yes, you can put beginning of stream after end-of-file without error.

## Reading from a single file

* f.read
* >>> f = open ('test.txt')
* >>> f
* <_io.TextIOWrapper name='test.txt' mode='r' encoding='UTF-8'> # file object
* >>> f.read # reads the whole file.
* 'Hello, world!\nHola, mundo!\nGoodbye, world!\n'
* >>> f.read
* '' # At end-of-file.
* >>> f.seek(0) # To read file again, put beginning of stream at beginning of file.
* 0
* >>> f.read
* 'Hello, world!\nHola, mundo!\nGoodbye, world!\n'
* >>> f.read
* '' # At end-of-file again.
* f.read may contain an optional argument size: f.read(size) where size is numeric, in
* which case at most size bytes are read and returned. The reference states: When size is omitted or negative, the entire contents of the file will be read and returned; it’s your problem if the file is twice as large as your machine’s memory.
* f.readline
* >>> f = open ('test.txt', mode='rt')
* >>> f.readline # Read one line of the file.
* 'Hello, world!\n' # '\n' at end has not been removed.
* >>> f.readline # Read next line of the file.
* 'Hola, mundo!\n'
* >>> f.readline # Read next line of the file.
* 'Goodbye, world!\n'
* >>> f.readline # Read next line of the file.
* '' # At end-of-file.
* To iterate using f.readline:
* f = open ('test.txt', mode='rt')
* while True :
* s = f.readline
* if s == '' : break
* print (s, end='')
* Hello, world!
* Hola, mundo!
* Goodbye, world!
* f.readline may contain an optional argument size: f.readline(size) where size is numeric, in
* which case at most size bytes are read and returned.
* f = open ('test.txt', mode='rt')
* while True :
* posn = f.tell
* s = f.readline(8)
* if s == '' : break
* print (" %2d %s" % (posn , s), end='')
* 0 Hello, w 8 orld!
* 14 Hola, mu 22 ndo!
* 27 Goodbye, 35 world!
* File object as iterable
* >>> f = open ('test.txt', mode='rt')
* >>> lines = list(f) ; lines
* ['Hello, world!\n', 'Hola, mundo!\n', 'Goodbye, world!\n']
* >>> lines = list(f) ; lines
* # At end-of-file
* >>> f.seek(0) # Put beginning of stream at beginning of file.
* 0
* >>> lines = list(f) ; lines
* ['Hello, world!\n', 'Hola, mundo!\n', 'Goodbye, world!\n']
* >>> lines = list(f) ; lines
* >>> f.seek(20) # Put beginning of stream at desired position.
* 20
* >>> list(f)
* ['mundo!\n', 'Goodbye, world!\n']
* >>> list(f)
* f = open('test.txt')
* for line in f : # This code is memory efficient,
* print (line, end='') # fast and simple.
* f.close
* Hello, world!
* Hola, mundo!
* Goodbye, world!
* Display all lines in the file that contain the word 'world':
* >>> f = open ('test.txt', mode='rt')
* >>> [p for p in f if 'world' in p]
* ['Hello, world!\n', 'Goodbye, world!\n']
* >>> f.close

## Reading international text

* File test1.txt contains:
* η ρωμαϊκή μυθολογία (Roman mythology)
* Всероссийская перепись населения 2010 года (2010 All-Russia Population Census)
* The last line is included to facilitate counting characters.
* Without knowing anything about Greek or Russian we see immediately that the Greek characters for iota ι
* ϊ ί
* are different, as are the Russian characters
* и й.
* The next thing to notice is that the file contains 249 bytes, but each line contains 38, 79, 81 characters or
* 198 characters for the whole file. Not to worry. In text files Python performs the appropriate encoding and decoding nicely:
* f = open('test1.txt')
* for line in f :
* print (len(line), line, end='')
* f.close
* 38 η ρωμαϊκή μυθολογία (Roman mythology)
* 79 Всероссийская перепись населения 2010 года (2010 All-Russia Population Census)
* 81 ..../...//..../...//..../...//..../...//..../...//..../...//..../...//..../...//
* The same again with different detail:
* f = open ('test1.txt', mode='rt')
* while True :
* posn = f.tell
* s = f.readline(30) # In text files read 30 characters.
* print (" %3d %s" % (posn , s), end='')
* if s == '' : break
* f.close
* 0 η ρωμαϊκή μυθολογία (Roman myt 47 hology)
* 55 Всероссийская перепись населен 113 ия 2010 года (2010 All-Russia 149 Population Census)
* 168 ..../...//..../...//..../...// 198 ..../...//..../...//..../...// 228 ..../...//..../...//
* 249
* Length of first line in bytes = 55. Length of first line in characters = 38.
* Length of second line in bytes = 168-55 = 113. Length of second line in characters = 79.
* Length of third line in bytes = 249-168 = 81. Length of third line in characters = 81.
* The first invocation of f.readline(30) read 30 characters (17 Greek and 13 English) in 47 (17*2 + 13) bytes,
* 47-30 = 17 extra bytes for 17 Greek characters.
* Similarly,
* the fourth invocation of f.readline(30) read 30 characters (6 Russian and 24 English) in 36 (149-113) or (6*2 + 24) bytes,
* 36-30 = 6 extra bytes for 6 Russian characters.
* When you're reading international text, the number of bytes read will be, almost certainly, more than the number of characters read.
* Take care if you reposition the stream into the middle of international text:
* f = open ('test1.txt', mode='rt')
* f.seek(4)
* posn = f.tell
* s = f.readline(30)
* print (" %3d %s" % (posn , s))
* Traceback (most recent call last):
* File "t3.py", line 11, in <module>
* s = f.readline(30)
* File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/codecs.py", line 321, in decode
* (result, consumed) = self._buffer_decode(data, self.errors, final)
* UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 0: invalid start byte

## Reading from multiple input streams

* The function fileinput.input(....) is the primary interface of this module:
* >>> import fileinput
* >>> fileinput.input( files=('test.txt', 'test1.txt') ) # The 2 named files provide the input stream.
* <fileinput.FileInput object at 0x101a96208>
* >>> list ( fileinput.input( files=('test.txt', 'test1.txt') ) )
* ['Hello, world!\n',
* 'Hola, mundo!\n',
* 'Goodbye, world!\n',
* 'η ρωμαϊκή μυθολογία (Roman mythology)\n',
* 'Всероссийская перепись населения 2010 года (2010 All-Russia Population Census)\n',
* '..../...//..../...//..../...//..../...//..../...//..../...//..../...//..../...//\n']
* # Output above was edited for clarity.
* Successive invocations of the stream do not require reopening or resetting the stream:
* >>> len(list ( fileinput.input( files=('test.txt', 'test1.txt') ) ))
* 6
* >>> len(list ( fileinput.input( files=('test.txt', 'test1.txt') ) ))
* 6
* >>> (list ( fileinput.input( files=('test.txt', 'test1.txt') ) ))[3]
* 'η ρωμαϊκή μυθολογία (Roman mythology)\n'
* >>> (list ( fileinput.input( files=('test.txt', 'test1.txt') ) ))[4]
* 'Всероссийская перепись населения 2010 года (2010 All-Russia Population Census)\n'
* >>> (list ( fileinput.input( files=('test.txt', 'test1.txt') ) ))[1]
* 'Hola, mundo!\n'
* >>> (list ( fileinput.input( files=('test.txt', 'test1.txt') ) ))[1]
* 'Hola, mundo!\n'
* >>> (list ( fileinput.input( files=('test.txt', 'test1.txt') ) ))[1:4]
* ['Hola, mundo!\n', 'Goodbye, world!\n', 'η ρωμαϊκή μυθολογία (Roman mythology)\n']
* The functions fileinput.filename, fileinput.lineno, fileinput.filelineno provide information about the stream:
* for line in fileinput.input( files=('test.txt', '/dev/null', 'test1.txt') ) :
* print ('filename =', fileinput.filename)
* print (' ', line, end='')
* print (' lineno =, filelineno =.'.format( fileinput.lineno, fileinput.filelineno ))
* exit (0)
* filename = test.txt
* Hello, world!
* lineno = 1, filelineno = 1.
* filename = test.txt
* Hola, mundo!
* lineno = 2, filelineno = 2.
* filename = test.txt
* Goodbye, world!
* lineno = 3, filelineno = 3.
* filename = test1.txt
* η ρωμαϊκή μυθολογία (Roman mythology)
* lineno = 4, filelineno = 1.
* filename = test1.txt
* Всероссийская перепись населения 2010 года (2010 All-Russia Population Census)
* lineno = 5, filelineno = 2.
* filename = test1.txt
* lineno = 6, filelineno = 3.
* The null file /dev/null was silently ignored.

## Writing text to a disk file

* The function f.write(str) may be used to write to a file in text mode or binary mode.
* In text mode it returns the number of characters written, in binary mode the number of bytes.
* s = '''η ρωμαϊκή μυθολογία (Greek characters)
* Всероссийская перепись (Russian or Cyrillic)
* The Quick BROWN foX (English characters)
* length_of_s = len(s) # in characters
* ofile = open('test1a.txt', 'w')
* number_written = ofile.write(s) # in characters
* end_of_file = ofile.tell # in bytes
* ofile.close
* print (
* length_of_s =
* number_written =
* end_of_file =
* '''.format(length_of_s, number_written, end_of_file)
* length_of_s = 125
* number_written = 125
* end_of_file = 163
* On the UNIX command line:
* $ wc test1a.txt
* 3 17 163 test1a.txt
* The UNIX executable wc (word count) shows that the output file contains 3 lines, 17 words and 163 bytes. The size in bytes agrees with end_of_file above.
* The file actually contains 16 words. wc sees 'μυθολογία' as 2 words, but Python performs the output without error.
* The difference between 163 and 125 (38) is because of the inclusion of 17 Greek letters and 21 Russian. Each of the international characters requires 2 bytes.

## Binary operations on disk files

* Brief review of binary conversion
* An int data type is conceptually a sequence of bytes. However, an int cannot be written to disk directly.
* Before an int can be written to disk in binary format, it must be converted to, eg, a bytes object or
* bytearray. This section illustrates the conversion to and from by means of examples.
* >>> b = 123456789
* >>> b1 = hex(b) ; b1
* '0x75bcd15'
* >>> b2 = b1[2:] ; b2
* '75bcd15'
* >>> b3 = '0'*(len(b2)%2) + b2 ; b3 # Ensure that the string contains an even number of hex digits.
* '075bcd15'
* >>> len(b3) % 2
* 0 # Length of b3 is even.
* >>> b4 = bytes.fromhex(b3) ; b4
* b'\x07[\xcd\x15' # b4 is a bytes object containing integer b in binary format.
* >>> isinstance(b4, bytes)
* True
* >>> list(b4)
* [7, 91, 205, 21]
* >>> 7 == b4[0] and 91 == b4[1] == ord('[') and 205 == b4[2] == 0xCD and 21 == b4[3] == 0x15
* True
* >>> (7<<24) + (91<<16) + (205<<8) + 21 == b # Check the conversion.
* True
* >>> # Convert from bytes object b4 to int:
* >>> b5 = b4.hex ; b5
* '075bcd15'
* >>> b6 = int(b5,16) ; b6
* 123456789
* >>> b6 == b
* True
* Methods int.to_bytes(length, ....) and int.from_bytes(bytes, ....)
* Method int.to_bytes(length, byteorder, *, signed=False) simplifies conversion from int to bytes.
* Class method int.from_bytes(bytes, byteorder, *, signed=False) simplifies the reverse.
* The following code ensures that the integer produced after encoding and decoding is the same as the original int:
* def int_to_bytes (input) : # input is int.
* num_bits = input.bit_length
* num_bytes = (num_bits + 7) // 8
* if ((num_bits % 8) == 0) : num_bytes += 1
* return input.to_bytes(num_bytes, byteorder='big', signed=True)
* def int_from_bytes (input) : # input is bytes.
* return int.from_bytes(input, byteorder='big', signed=True)
* Writing to disk in binary mode
* Integer i1 contains binary data. Convert i1 to bytes and write to disk.
* i1 = 123456789012345678901234567890123456789012345678901234567890
* b1 = int_to_bytes (i1)
* b1a = r"""b'\x13\xaa\xf5\x04\xe4\xbc\x1e\x62\x17\x3f\x87\xa4\x37\x8c\x37\xb4\x9c\x8c\xcf\xf1\x96\xce\x3f\x0a\xd2'"""
* print (
* r'''
* i1 = # original int
* b1 = # original int as bytes
* b1a is b1 expanded so that each byte is expressed as '\xHH':
* b1a =
* b1 == b1a : # b1a matches the hex representation of i1.
* '''.format( hex(i1), b1, b1a, b1 == eval(b1a) ) # b1a is str.
* try:
* with open("test.bin", "wb") as ofile:
* number_written = ofile.write(b1) # in bytes
* end_of_file = ofile.tell # in bytes
* except:
* print ('Error detected in "with" statement.')
* print (
* len(b1) =
* number_written =
* end_of_file =
* .format(len(b1), number_written, end_of_file)
* i1 = 0x13aaf504e4bc1e62173f87a4378c37b49c8ccff196ce3f0ad2 # original int
* b1 = b'\x13\xaa\xf5\x04\xe4\xbc\x1eb\x17?\x87\xa47\x8c7\xb4\x9c\x8c\xcf\xf1\x96\xce?\n\xd2' # original int as bytes
* b1a is b1 expanded so that each byte is expressed as '\xHH':
* b1a = b'\x13\xaa\xf5\x04\xe4\xbc\x1e\x62\x17\x3f\x87\xa4\x37\x8c\x37\xb4\x9c\x8c\xcf\xf1\x96\xce\x3f\x0a\xd2'
* b1 == b1a : True # b1a matches the hex representation of i1.
* len(b1) = 25
* number_written = 25
* end_of_file = 25
* $ od -t x1 test.bin
* 0000000 13 aa f5 04 e4 bc 1e 62 17 3f 87 a4 37 8c 37 b4
* 0000020 9c 8c cf f1 96 ce 3f 0a d2
* 0000031
* Reading from disk in binary mode
* The file on disk contains a large int in bytes format. Read the file from disk and convert to int.
* try:
* with open("test.bin", "rb") as ifile:
* b2 = ifile.read
* end_of_file = ifile.tell # in bytes
* except:
* print ('Error detected in "with" statement.')
* i2 = int_from_bytes (b2)
* print (
* isinstance(b2, bytes):
* len(b2) =
* end_of_file =
* i2 == i1:
* '''.format( isinstance(b2, bytes), len(b2), end_of_file, i2 == i1)
* isinstance(b2, bytes): True
* len(b2) = 25
* end_of_file = 25
* i2 == i1: True
* Reading text in binary mode
* Data may be read from a text file in binary mode. The disadvantage is that python
* does not automatically perform the necessary decoding.
* try:
* with open("test1a.txt", "rb") as ifile:
* b5 = ifile.read
* end_of_file = ifile.tell # in bytes
* except:
* print ('Error detected in "with" statement.')
* b6 = b5.decode(encoding='utf-8')
* print (
* isinstance(b5, bytes):
* len(b5) = bytes
* end_of_file =
* b5 = (
* b5 decoded =
* length of b5 decoded = characters
* '''.format( isinstance(b5, bytes), len(b5), end_of_file,
* b5[:18] , b5[18:56] , b5[56:83] , b5[83:122] , b5[122:] ,
* b6, len(b6) )
* isinstance(b5, bytes): True
* len(b5) = 163 bytes
* end_of_file = 163
* b5 = (
* b'\xce\xb7 \xcf\x81\xcf\x89\xce\xbc\xce\xb1\xcf\x8a\xce\xba\xce\xae '
* + b'\xce\xbc\xcf\x85\xce\xb8\xce\xbf\xce\xbb\xce\xbf\xce\xb3\xce\xaf\xce\xb1 (Greek characters)\n'
* + b'\xd0\x92\xd1\x81\xd0\xb5\xd1\x80\xd0\xbe\xd1\x81\xd1\x81\xd0\xb8\xd0\xb9\xd1\x81\xd0\xba\xd0\xb0\xd1\x8f '
* + b'\xd0\xbf\xd0\xb5\xd1\x80\xd0\xb5\xd0\xbf\xd0\xb8\xd1\x81\xd1\x8c (Russian or Cyrillic)\n'
* + b'The Quick BROWN foX (English characters)\n'
* b5 decoded =
* η ρωμαϊκή μυθολογία (Greek characters)
* Всероссийская перепись (Russian or Cyrillic)
* The Quick BROWN foX (English characters)
* length of b5 decoded = 125 characters
* Seeking in binary mode
* f.seek works as expected in binary mode:
* >>> f = open("test.bin", "rb")
* >>> f.seek(0,2) # seek to end-of-file.
* 25 # File contains 25 bytes.
* >>> f.tell
* 25
* >>> f.seek(-3,0) # You cannot seek to a position before beginning of file.
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* OSError: [Errno 22] Invalid argument
* >>> f.tell
* 25 # Still at end-of-file.
* >>> f.seek(3,0) # relative to beginning of file.
* 3
* >>> f.seek(4,1) # relative to current position.
* 7
* >>> f.seek(-5,2) # relative to end-of-file.
* 20
* >>> f.seek(123456)
* 123456 # Yes, you can position the internal pointer to a position after end-of-file.
* >>> f.close
* Truncating a file in binary mode
* Function os.ftruncate(fd, length) uses a file descriptor.
* import os
* try:
* with open("test.bin", "r+b") as f:
* print ('File is writable:', f.writable)
* print ('File is readable:', f.readable)
* b = f.read
* if len(b) != f.tell : raise OSError
* print ('b =', b, '# Original contents.')
* print ('Truncate the file to current size - 10')
* os.ftruncate(f.fileno, f.seek(0,2) - 10)
* f.seek(0)
* b = f.read
* if len(b) != f.tell : raise OSError
* print ('b =', b, '# Last 10 bytes removed.')
* print ('Truncate the file to current size + 10')
* os.ftruncate(f.fileno, f.seek(0,2) + 10)
* f.seek(0)
* b = f.read
* if len(b) != f.tell : raise OSError
* print ('b =', b, '# 10 null bytes added at end.')
* print ('Truncate the file to 4 bytes')
* os.ftruncate(f.fileno, 4)
* f.seek(0)
* b = f.read
* if len(b) != f.tell : raise OSError
* print ('b =', b, '# Truncated to 4 bytes.')
* except OSError:
* print ('OSError detected in "with" statement.')
* except FileNotFoundError:
* print ('FileNotFoundError detected in "with" statement.')
* except PermissionError:
* print ('PermissionError detected in "with" statement.')
* except:
* print ('Unknown error detected in "with" statement.')
* File is writable: True
* File is readable: True
* b = b'\x13\xaa\xf5\x04\xe4\xbc\x1eb\x17?\x87\xa47\x8c7\xb4\x9c\x8c\xcf\xf1\x96\xce?\n\xd2' # Original contents.
* Truncate the file to current size - 10
* b = b'\x13\xaa\xf5\x04\xe4\xbc\x1eb\x17?\x87\xa47\x8c7' # Last 10 bytes removed.
* Truncate the file to current size + 10
* b = b'\x13\xaa\xf5\x04\xe4\xbc\x1eb\x17?\x87\xa47\x8c7\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' # 10 null bytes added at end.
* Truncate the file to 4 bytes
* b = b'\x13\xaa\xf5\x04' # Truncated to 4 bytes.

## External operations on files

* External operations are similar to those executed on the Unix command line.
* They do not depend on having a file open before execution:
* Creating a file
* On the Unix command line:
* $ ls -la test1.txt test5.txt
* ls: test5.txt: No such file or directory
* -rw-r--r-- 1 user staff 249 Sep 15 06:53 test1.txt
* >>> f = open ('test1.txt', 'x') # 'x' for exclusive creation.
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* FileExistsError: [Errno 17] File exists: 'test1.txt'
* >>> f
* <_io.BufferedRandom name='test1.bin'> # Unchanged. f refers to previous open.
* >>> f = open ('test5.txt', 'x')
* >>> f
* <_io.TextIOWrapper name='test5.txt' mode='x' encoding='UTF-8'>
* >>> f.closed
* False
* >>> f.readable
* False
* >>> f.writable
* True
* >>> f.close
* On the Unix command line:
* $ ls -la test1.txt test5.txt
* -rw-r--r-- 1 user staff 249 Sep 15 06:53 test1.txt
* -rw-r--r-- 1 user staff 0 Oct 4 08:28 test5.txt # File was created.
* Truncating a file
* >>> os.stat ('test1.txt').st_size
* 249
* >>> os.truncate('test1.txt', 200)
* >>> os.stat ('test1.txt').st_size
* 200
* >>> os.stat ('test5.txt').st_size
* 0
* >>> os.truncate('test5.txt', 150)
* >>> os.stat ('test5.txt').st_size
* 150 # 150 bytes were added to file.
* On the Unix command line:
* $ ls -la test1.txt test5.txt
* -rw-r--r-- 1 user staff 200 Oct 4 08:37 test1.txt
* -rw-r--r-- 1 user staff 150 Oct 4 08:38 test5.txt
* $ od -h test5.txt
* 0000000 0000 0000 0000 0000 0000 0000 0000 0000
* 0000220 0000 0000 0000
* 0000226
* $ # test5.txt contains 150 null bytes after truncation.
* Accessing a file
* os.access(path, mode, ....) may be used to determine the existence, readability, writability and executability of path.
* $ ls -la t*n
* --w-r----- 1 user staff 25 Sep 29 13:12 test.bin
* -rw-r--r-- 1 user staff 25 Sep 29 08:03 test1.bin
* >>> os.access('test0.txt', os.F_OK)
* False # The file does not exist.
* >>> os.access('test.bin', os.F_OK)
* True # The file exists.
* >>> os.access('test.bin', os.R_OK)
* False # It is not readable.
* >>> os.access('test.bin', os.W_OK)
* True # It is writable.
* >>> os.access('test.bin', os.X_OK)
* False # It is not executable.
* >>> os.access('test1.bin', os.F_OK + os.R_OK + os.W_OK) # Modes may be added.
* True # The file exists and it's both readable and writable.
* Changing a file's u,g,o permissions
* os.chmod(path, mode, *, dir_fd=None, follow_symlinks=True) may be used to change the permissions for user, group and other.
* On the Unix command line:
* $ ls -la t*n
* --w-r----- 1 user staff 25 Sep 29 13:12 test.bin
* -rw-r--r-- 1 user staff 25 Sep 29 08:03 test1.bin
* On the python command line:
* >>> import os
* >>> import stat
* >>> os.chmod('test.bin', stat.S_IRWXG) # read, write, execute for group.
* On the Unix command line:
* $ ls -la t*n
* ----rwx--- 1 user staff 25 Sep 29 13:12 test.bin # Permissions changed to r,w,x for group.
* -rw-r--r-- 1 user staff 25 Sep 29 08:03 test1.bin
* Renaming a file
* os.rename(src, dst, *, src_dir_fd=None, dst_dir_fd=None) may be used to rename a file.
* On the Unix command line:
* $ ls -la test1*t
* -rw-r--r-- 1 user staff 200 Oct 4 08:37 test1.txt
* -rw-r--r-- 1 user staff 163 Sep 18 18:07 test1a.txt
* On the python command line:
* >>> os.rename('test1.txt', 'test1a.txt')
* On the Unix command line:
* $ ls -la test1*t
* -rw-r--r-- 1 user staff 200 Oct 4 08:37 test1a.txt
* The old file test1a.txt was silently deleted.
* os.rename(src, dst, ....) and os.replace(src, dst, ....) are almost identical with slight differences dependent on Operating System.
* Removing a file
* os.remove(path, *, dir_fd=None) may be used to remove (delete) a file.
* On the Unix command line:
* $ ls -la test1*t
* -rw-r--r-- 1 user staff 200 Oct 4 08:37 test1a.txt
* On the python command line:
* >>> os.remove('test1a.txt')
* On the Unix command line:
* $ ls -la test1*t
* ls: test1*t: No such file or directory
* The file test1a.txt was deleted.
* This function and unlink are semantically identical.

## Objects that behave like files

* The terminal
* On Unix each terminal window has its unique device name, a name that looks like a file name, eg, /dev/ttys003.
* Communication with the console may be achieved by treating the console like a file:
* $ cat t5.py
* import os
* print ('Name of my terminal is:', os.ttyname(0))
* f = open(os.ttyname(0), 'wt')
* print ('File object opened for writing to my terminal is:', f)
* print ('Enter your date-of-birth [mm/dd/yyyy]: ', end='', flush=True, file=f) # Writing to terminal.
* f.close
* f = open(os.ttyname(0))
* dob = f.read # Reading from terminal.
* print ('File object opened for reading from my terminal is:', f)
* f.close
* print ('You entered:', dob, end='')
* $ python3.6 t5.py
* Name of my terminal is: /dev/ttys003
* File object opened for writing to my terminal is: <_io.TextIOWrapper name='/dev/ttys003' mode='wt' encoding='UTF-8'>
* Enter your date-of-birth [mm/dd/yyyy]: 12/31/1999 # Enter dob followed by new-line and ^D for end-of-file.
* File object opened for reading from my terminal is: <_io.TextIOWrapper name='/dev/ttys003' mode='r' encoding='UTF-8'>
* You entered: 12/31/1999
* Pipes
* os.pipe creates a pipe:
* >>> fdr,fdw = os.pipe
* >>> fdr
* 3 # The file descriptor for reading from the pipe.
* >>> fdw
* 4 # The file descriptor for writing to the pipe.
* >>> os.write(fdw, b'Hello, world!') # Write to the file descriptor associated with the pipe.
* 13 # Number of bytes written.
* >>> os.read(fdr, 7) # Read a max of 7 bytes from the file descriptor associated with the pipe.
* b'Hello, '
* >>> os.read(fdr, 99) # Read a max of 99 bytes from fdr.
* b'world!'
* >>> os.close(fdr)
* >>> os.close(fdw)
* Standard input is usually file descriptor 0, standard output is 1, and standard error is 2. Further files opened by a process will then be assigned 3, 4, 5, and so forth. Hence file descriptors 3 and 4 above.
* The pipe implements a fifo, first-in-first-out queue. Data added to the pipe is appended to the data in the pipe. Data removed from the pipe is read from the beginning of data in the pipe.
* Before trying to read data from a pipe ensure that there is data in the pipe:
* import os
* fdr,fdw = os.pipe
* number_written = os.write(fdw, b'9876543210')
* number_of_bytes_in_pipe = os.stat(fdr).st_size
* if number_written == number_of_bytes_in_pipe == 10 :
* pass # Integrity of data looks good.
* else :
* print ('Internal error: Input data corrupted.')
* exit (99)
* print ('number_of_bytes_in_pipe =', number_of_bytes_in_pipe)
* number_of_bytes_read_from_pipe = 0
* while os.stat(fdr).st_size > 0 :
* data = os.read(fdr,1)
* number_of_bytes_read_from_pipe += len(data)
* print ('data =', data)
* os.close(fdr)
* os.close(fdw)
* print ('number_of_bytes_read_from_pipe =', number_of_bytes_read_from_pipe)
* exit (0)
* number_of_bytes_in_pipe = 10
* data = b'9'
* data = b'8'
* data = b'7'
* data = b'6'
* data = b'5'
* data = b'4'
* data = b'3'
* data = b'2'
* data = b'1'
* data = b'0'
* number_of_bytes_read_from_pipe = 10
* It seems that functionsos.lseek(fd, ....), os.pread(fd, ....) do not work with pipes.

## File objects and file descriptors

* On the Unix command line:
* $ ls -l test.txt ; cat test.txt
* -rw-r--r-- 1 user staff 43 Sep 9 17:26 test.txt
* Hello, world!
* Hola, mundo!
* Goodbye, world!
* The function open returns a file object and silently creates a file descriptor:
* >>> f = open('test.txt')
* >>> f
* <_io.TextIOWrapper name='test.txt' mode='r' encoding='UTF-8'>
* >>> fd_f = f.fileno # fd_f (an int) is the file descriptor associated with f.
* >>> fd_f
* 5
* >>> os.stat(fd_f).st_size # The file descriptor can provide info about the file.
* 43 # Size of file 'test.txt' in bytes
* >>> os.lseek(fd_f,10,0) # Seek to beginning-of-file + 10
* 10
* >>> f.tell
* 10
* >>> os.lseek(fd_f,-3,1) # Seek -3 relative to current position.
* 7 # Using fd_f it works.
* >>> f.seek(-3,1) # Using f it doesn't work in text mode.
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* io.UnsupportedOperation: can't do nonzero cur-relative seeks
* >>> f.tell
* 7
* >>> f.seek(14)
* 14
* >>> os.lseek(fd_f,0,1) # Report current position.
* 14
* >>> f.read(13)
* 'Hola, mundo!\n'
* >>> f.tell
* 27
* >>> os.lseek(fd_f,0,1)
* 43 # Internal pointers of f and fd_f are NOT always the same.
* >>> os.lseek(fd_f,-16,2) # Seek -16 relative to end-of-file.
* 27
* >>> os.read(fd_f,99)
* b'Goodbye, world!\n'
* >>> f.closed
* False
* >>> f.close # This also closes fd_f.
* >>> f.closed
* True
* >>> os.stat(fd_f).st_size
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* OSError: [Errno 9] Bad file descriptor: 5
* Multiple file objects with same file descriptor
* Two or more file objects may have the same file descriptor:
* >>> f = open('test.txt')
* >>> f1 = open(f.fileno, 'w', closefd=False)
* >>> f
* <_io.TextIOWrapper name='test.txt' mode='r' encoding='UTF-8'>
* >>> f1
* <_io.TextIOWrapper name=5 mode='w' encoding='UTF-8'>
* >>> f.fileno
* 5
* >>> f1.fileno
* 5
* >>> f.closed
* False
* >>> f1.closed
* False
* >>> f.close
* >>> f.closed
* True
* >>> f1.closed
* False # f1 remains open
* >>> f1.fileno
* 5
* >>> f1.readable
* False
* >>> f1.writable
* True # This info is deceptive.
* If you have two file objects associated with the same file descriptor and you close one of the file objects, the behavior of the other may be
* unpredictable. Unless you really know what you're doing, when you close one file object of many associated with the same file descriptor close
* them all. Also, don't close the file descriptor (with os.close(fd))
* before closing the file object. This creates a really messy situation.

## Temporary files

* python's tempfile module contains functions that can be used to generate temporary files.
* Depending on the function and parameters used, the file created may or may not be visible on the file system,
* it may or may not be deleted
* when the file is closed, and it may or may not be opened in binary mode.
* Function tempfile.NamedTemporaryFile(....) is representative of the functions available for file creation in module tempfile:
* tempfile.NamedTemporaryFile(
* mode='w+b', # For consistent behavior across platforms.
* buffering=None,
* encoding=None, # Buffering, encoding and newline are as for open.
* newline=None,
* suffix='', # With 'suffix' and 'prefix' you have some control over the file name.
* prefix='tmp',
* dir=None, # You can specify the directory where the file will be created.
* delete=True # You can choose to keep the file after closure.
* Opening a temporary file in text mode for deletion on closing:
* >>> import tempfile
* >>> f1 = tempfile.NamedTemporaryFile(mode='r+t', suffix='.txt')
* >>> f1
* <tempfile._TemporaryFileWrapper object at 0x101b7ab38>
* >>> isinstance(f1, tempfile._TemporaryFileWrapper)
* True
* >>> f1.name
* '/var/folders/8s/dctgn1y57fgc9_h2mzckbqs80000gn/T/tmpew4jippj.txt'
* >>> f1.readable
* True
* >>> f1.writable
* True
* >>> f1.seekable
* True
* >>> f1.encoding
* 'UTF-8'
* >>> f1.write('Hello, world!\n')
* 14
* >>> f1.write('Καλώς ήρθατε στο Βικιεπιστήμιο\n')
* 31 # 27 Greek + 4 English characters = 27*2 + 4 = 58 bytes.
* >>> f1.tell
* 72 # 14 + 58
* >>> f1.seek(0)
* 0
* >>> f1.readline
* 'Hello, world!\n'
* >>> f1.readline
* 'Καλώς ήρθατε στο Βικιεπιστήμιο\n'
* >>> f1.readline
* >>> os.stat( f1.name )
* os.stat_result(st_mode=33152, st_ino=24460790, st_dev=16777218, st_nlink=1, st_uid=501, st_gid=20, st_size=72, st_atime=1508154874, st_mtime=1508154545, st_ctime=1508154545)
* >>> f1.close
* >>> f1.name
* '/var/folders/8s/dctgn1y57fgc9_h2mzckbqs80000gn/T/tmpew4jippj.txt'
* >>> os.stat( f1.name )
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* FileNotFoundError: [Errno 2] No such file or directory:
* '/var/folders/8s/dctgn1y57fgc9_h2mzckbqs80000gn/T/tmpew4jippj.txt'
* Opening a temorary file in binary mode for retention on closing:
* >>> f1 = tempfile.NamedTemporaryFile(suffix='.bin', delete=False)
* >>> f1
* <tempfile._TemporaryFileWrapper object at 0x101a8c518>
* >>> f1.write( b'When in the course of ....' )
* 26
* >>> f1.name
* '/var/folders/8s/dctgn1y57fgc9_h2mzckbqs80000gn/T/tmpdz14i44i.bin'
* >>> f1.tell
* 26
* >>> f1.seek(0)
* 0
* >>> f1.read
* b'When in the course of ....'
* >>> f1.close
* The temporary file exists after closing:
* $ ls -la /var/folders/8s/dctgn1y57fgc9_h2mzckbqs80000gn/T/tmpdz14i44i.bin
* -rw------- 1 user staff 26 Oct 16 07:22 /var/folders/8s/dctgn1y57fgc9_h2mzckbqs80000gn/T/tmpdz14i44i.bin

## Further Reading or Review

* Previous Lesson: Console Input
* This Lesson:Files
* Next Lesson: Directories
* Course Home Page