markhneedham/blog huggingface dataset:
Python: Parsing a JSON HTTP chunking stream
Date: 2015-11-28 13:56:59

I've been playing around with meetup.com's API again and this time wanted to consume the http://www.meetup.com/meetup_api/docs/stream/2/rsvps/#http[chunked HTTP RSVP stream] and filter RSVPs for events I'm interested in.

I use Python for most of my hacking these days and if HTTP requests are required the http://docs.python-requests.org/en/latest/[requests library] is my first port of call.

I started out with the following script

[source,python]
----

import requests
import json

def stream_meetup_initial():
    uri = "http://stream.meetup.com/2/rsvps"
    response = requests.get(uri, stream = True)
    for chunk in response.iter_content(chunk_size = None):
        yield chunk

for raw_rsvp in stream_meetup_initial():
    print raw_rsvp
    try:
        rsvp = json.loads(raw_rsvp)
    except ValueError as e:
        print e
        continue
----

This mostly worked but I also noticed the following error from time to time:

[source,text]
----

No JSON object could be decoded
----

Although less frequent, I also saw errors suggesting I was trying to parse an incomplete JSON object. I tweaked the function to keep a local buffer and only yield that if the chunk ended in a new line character:

[source,python]
----

def stream_meetup_newline():
    uri = "http://stream.meetup.com/2/rsvps"
    response = requests.get(uri, stream = True)
    buffer = ""
    for chunk in response.iter_content(chunk_size = 1):
        if chunk.endswith("\n"):
            buffer += chunk
            yield buffer
            buffer = ""
        else:
            buffer += chunk
----

This mostly works although I'm sure I've seen some occasions where two JSON objects were being yielded and then the call to 'json.loads' failed. I haven't been able to reproduce that though.

A second read through the http://docs.python-requests.org/en/latest/user/advanced/#streaming-requests[requests documentation] made me realise I hadn't read it very carefully the first time since we can make our lives much easier by using 'iter_lines' rather than 'iter_content':

[source,python]
----

r = requests.get('http://stream.meetup.com/2/rsvps', stream=True)
for raw_rsvp in r.iter_lines():
    if raw_rsvp:
        rsvp = json.loads(raw_rsvp)
        print rsvp
----

We can then process 'rsvp', filtering out the ones we're interested in.
Docker 1.9: Port forwarding on Mac OS X
Date: 2015-11-08 20:58:42

Since the http://www.infoq.com/news/2015/10/neo4j-2.3-release[Neo4j 2.3.0 release] there's been an https://hub.docker.com/r/neo4j/neo4j/[official docker image] which I thought I'd give a try this afternoon.

The last time I used docker about a year ago I had to install http://boot2docker.io/[boot2docker] which has now been deprecated in place of http://docs.docker.com/engine/installation/mac/[Docker Machine and the Docker Toolbox].

I created a container with the following command:

[source,bash]
----

docker run --detach --publish=7474:7474 neo4j/neo4j
----

And then tried to access the Neo4j server locally:

[source,bash]
----

$ curl http://localhost:7474
curl: (7) Failed to connect to localhost port 7474: Connection refused
----

I quickly checked that docker had started up Neo4j correctly:

[source,bash]
----

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                              NAMES
1f7c48e267f0        neo4j/neo4j         "/docker-entrypoint.s"   10 minutes ago      Up 10 minutes       7473/tcp, 0.0.0.0:7474->7474/tcp   kickass_easley
----

Looks good. Amusingly I then came across http://www.markhneedham.com/blog/2014/11/27/dockerneo4j-port-forwarding-on-mac-os-x-not-working/[my own blog post from a year ago] where I'd run into the same problem - the problem being that we need to access the Neo4j server via the VM's IP address rather than localhost.

Instead of using boot2docker we now need to use docker-machine to find the VM's IP address:

[source,bash]
----

$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM
default   *        virtualbox   Running   tcp://192.168.99.100:2376
----

[source,bash]
----

$ curl http://192.168.99.100:7474
{
  "management" : "http://192.168.99.100:7474/db/manage/",
  "data" : "http://192.168.99.100:7474/db/data/"
}
----

And we're back in business.
Mac OS X: Installing the PROJ.4 - Cartographic Projections Library
Date: 2015-10-05 22:41:10

I've been following http://scottbarnham.com/blog/2010/08/27/uk-postcodes-to-latitudelongitude/[Scott Barnham's guide to transforming UK postcodes into (lat, long) coordinates] and needed to install the https://github.com/OSGeo/proj.4[PROJ.4 Cartographic Projections library] which I initially struggled with.

The first step is to download a tar.gz version which is linked from the https://github.com/OSGeo/proj.4/wiki[wiki page]:

[source,bash]
----

$ wget http://download.osgeo.org/proj/proj-4.9.1.tar.gz
----

Next we'll unpack the file and then build the binaries:

[source,bash]
----

$ tar -xvf proj-4.9.1.tar.gz
$ cd proj-4.9.1
$ ./configure --prefix ~/projects/land-registry/proj-4.9.1
$ make
$ make install
----

The files we need are in the bin directory\...

[source,bash]
----

$ ls -alh bin/
total 184
drwxr-xr-x   8 markneedham  staff   272B  5 Oct 23:07 .
drwxr-xr-x@ 41 markneedham  staff   1.4K  5 Oct 20:46 ..
-rwxr-xr-x   1 markneedham  staff    20K  5 Oct 23:07 cs2cs
-rwxr-xr-x   1 markneedham  staff    16K  5 Oct 23:07 geod
lrwxr-xr-x   1 markneedham  staff     4B  5 Oct 23:07 invgeod -> geod
lrwxr-xr-x   1 markneedham  staff     4B  5 Oct 23:07 invproj -> proj
-rwxr-xr-x   1 markneedham  staff    13K  5 Oct 23:07 nad2bin
-rwxr-xr-x   1 markneedham  staff    21K  5 Oct 23:07 proj
----

\...now let's give it a try. We need to feed in OSGB36 grid reference values and then we'll get back WGS84 Lat/Lng values. We can grab some grid reference values from the http://www.ordnancesurvey.co.uk/oswebsite/products/code-point-open/index.html[Ordnance Survey website].

e.g. the Neo4j London office has the post code SE1 0NZ which translates to coordinates 531950,180195. Let's try those out with PROJ.4:

[source,bash]
----

$ ./proj-4.9.1/bin/cs2cs -f '%.7f' +proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +towgs84=446.448,-125.157,542.060,0.1502,0.2470,0.8421,-20.4894 +units=m +no_defs +to +proj=latlong +ellps=WGS84 +towgs84=0,0,0 +nodefs

531950 180195
-0.1002020	51.5052917 46.0810195
----

So it's suggested a (lat, long) pairing of (51.5052917, -0.1002020). And if we plug that into https://www.google.co.uk/maps/place/51%C2%B030'19.1%22N+0%C2%B006'00.7%22W/@51.5056056,-0.1011247,17z/data=!4m2!3m1!1s0x0:0x0[Google maps]\...

image::{{<siteurl>}}/uploads/2015/10/2015-10-05_23-29-09.png[2015 10 05 23 29 09,400]

\...it's pretty much spot on!
R: data.table - Finding the maximum row
Date: 2015-10-02 18:42:47

In my continued playing around with the R https://cran.r-project.org/web/packages/data.table/index.html[data.table] package I wanted to find the maximum row based on one of the columns, grouped by another column, and then return back the whole row.

We'll use the following data table to illustrate:

[source,r]
----

> blogDT = data.table(name = c("Property 1","Property 1","Property 1","Property 2","Property 2","Property 2"),
                    price = c(10000, 12500, 18000, 245000, 512000, 1000000),
                    date = c("Day 1", "Day 7", "Day 10", "Day 3", "Day 5", "Day 12"))

> blogDT[, lag.price := c(NA, price[-.N]), by = name]

> blogDT[, diff := price - lag.price]

> blogDT
         name   price   date lag.price   diff
1: Property 1   10000  Day 1        NA     NA
2: Property 1   12500  Day 7     10000   2500
3: Property 1   18000 Day 10     12500   5500
4: Property 2  245000  Day 3        NA     NA
5: Property 2  512000  Day 5    245000 267000
6: Property 2 1000000 Day 12    512000 488000
----

I wanted to find the biggest difference in 'price' and 'lag.price' grouped by the 'name' column.

If we just want to get the max 'diff' grouped by 'name' it's quite easy:

[source,r]
----

> blogDT[!is.na(diff), .(max = max(diff)), keyby = name]
         name    max
1: Property 1   5500
2: Property 2 488000
----

However now we've lost the information about which 'date' that was on and what the 'price' and 'lag.price' were which is a bit annoying.

If we only want to keep the rows which have the highest 'diff' grouped by 'name', one way to go about this is to add a 'max.diff' column to each row and then filter appropriately. e.g.

[source,r]
----

> maxDT = blogDT[!is.na(diff)][, max := max(diff), by = name]

> maxDT
         name   price   date lag.price   diff    max
1: Property 1   12500  Day 7     10000   2500   5500
2: Property 1   18000 Day 10     12500   5500   5500
3: Property 2  512000  Day 5    245000 267000 488000
4: Property 2 1000000 Day 12    512000 488000 488000

> maxDT[diff == max]
         name   price   date lag.price   diff    max
1: Property 1   18000 Day 10     12500   5500   5500
2: Property 2 1000000 Day 12    512000 488000 488000
----

I've only been playing with data.table for a few days so if there's a better way do let me know in the comments.
R: Calculating the difference between ordered factor variables
Date: 2015-07-02 22:55:01

In my continued exploration of https://github.com/mneedham/neo4j-wimbledon/blob/master/wimbledon.csv[Wimbledon data] I wanted to work out whether a player had done as well as their seeding suggested they should.

I therefore wanted to work out the difference between the round they reached and the round they were expected to reach. A 'round' in the dataset is an ordered factor variable.

These are all the possible values:

[source,r]
----

rounds = c("Did not enter", "Round of 128", "Round of 64", "Round of 32", "Round of 16", "Quarter-Finals", "Semi-Finals", "Finals", "Winner")
----

And if we want to factorise a couple of strings into this factor we would do it like this:

[source,r]
----

round = factor("Finals", levels = rounds, ordered = TRUE)
expected = factor("Winner", levels = rounds, ordered = TRUE)

> round
[1] Finals
9 Levels: Did not enter < Round of 128 < Round of 64 < Round of 32 < Round of 16 < Quarter-Finals < ... < Winner

> expected
[1] Winner
9 Levels: Did not enter < Round of 128 < Round of 64 < Round of 32 < Round of 16 < Quarter-Finals < ... < Winner
----

In this case the difference between the actual round and expected round should be -1 - the player was expected to win the tournament but lost in the final. We can calculate that differnce by http://stackoverflow.com/questions/7611810/converting-a-factor-to-numeric-without-losing-information-r-as-numeric-doesn[calling the +++<cite>+++unclass+++</cite>+++ function on each variable]:

[source,r]
----


> unclass(round) - unclass(expected)
[1] -1
attr(,"levels")
[1] "Did not enter"  "Round of 128"   "Round of 64"    "Round of 32"    "Round of 16"    "Quarter-Finals"
[7] "Semi-Finals"    "Finals"         "Winner"
----

That still seems to have some remnants of the factor variable so to get rid of that we can cast it to a numeric value:

[source,r]
----

> as.numeric(unclass(round) - unclass(expected))
[1] -1
----

And that's it! We can now go and apply this calculation to all seeds to see how they got on.
Python: Difference between two datetimes in milliseconds
Date: 2015-07-28 20:05:47

I've been doing a bit of adhoc measurement of some cypher queries executed via http://py2neo.org/2.0/[py2neo] and wanted to work out how many milliseconds each query was taking end to end.

I thought there'd be an obvious way of doing this but if there is it's evaded me so far and I ended up http://stackoverflow.com/questions/1345827/how-do-i-find-the-time-difference-between-two-datetime-objects-in-python[calculating the different between two _datetime_ objects] which gave me the following timedelta object: ~~~python >>> import datetime >>> start = datetime.datetime.now() >>> end = datetime.datetime.now() >>> end - start datetime.timedelta(0, 3, 519319) ~~~

The 3 parts of this object are 'days', 'seconds' and 'microseconds' which I found quite strange!

These are the methods/attributes we have available to us: ~~~python >>> dir(end - start) ['__abs__', '__add__', '__class__', '__delattr__', '__div__', '__doc__', '__eq__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__mul__', '__ne__', '__neg__', '__new__', '__nonzero__', '__pos__', '__radd__', '__rdiv__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rmul__', '__rsub__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', 'days', 'max', 'microseconds', 'min', 'resolution', 'seconds', 'total_seconds'] ~~~

There's no 'milliseconds' on there so we'll have to calculate it from what we do have: ~~~python >>> diff = end - start >>> elapsed_ms = (diff.days * 86400000) + (diff.seconds * 1000) + (diff.microseconds / 1000) >>> elapsed_ms 3519 ~~~

Or we could do the following slightly simpler calculation: ~~~python >>> diff.total_seconds() * 1000 3519.319 ~~~

And now back to the query profiling!
Python: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
Date: 2015-07-15 06:20:07

I was recently doing some text scrubbing and had difficulty working out how to remove the '†' character from strings.

e.g. I had a string like this:

[source,python]
----

>>> u'foo †'
u'foo \u2020'
----

I wanted to get rid of the '†' character and then strip any trailing spaces so I'd end up with the string 'foo'. I tried to do this in one call to 'replace':

[source,python]
----

>>> u'foo †'.replace(" †", "")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1: ordinal not in range(128)
----

It took me a while to work out that "† " was being treated as ASCII rather than UTF-8. Let's fix that:

[source,python]
----

>>> u'foo †'.replace(u' †', "")
u'foo'
----

I think the following call to +++<cite>+++unicode+++</cite>+++, which I've http://www.markhneedham.com/blog/2015/05/21/python-unicodeencodeerror-ascii-codec-cant-encode-character-uxfc-in-position-11-ordinal-not-in-range128/[written about before], is equivalent:

[source,python]
----

>>> u'foo †'.replace(unicode(' †', "utf-8"), "")
u'foo'
----

Now back to the scrubbing!
R: data.table - Comparing adjacent rows
Date: 2015-09-27 22:02:07

As part of my exploration of the https://data.gov.uk/dataset/land-registry-monthly-price-paid-data[Land Registry price paid data set] I wanted to compare the difference between consecutive sales of properties.

This means we need to group the sales by a property identifier and then get the previous sale price into a column on each row unless it's the first sale in which case we'll have 'NA'. We can do this by creating a +++<cite>+++http://stackoverflow.com/questions/26291988/r-how-to-create-a-lag-variable-for-each-by-group[lag]+++</cite>+++ variable.

I'll use a simpler data set which is very similar in structure to the Land Registry's to demonstrate:

[source,r]
----

> blogDT = data.table(name = c("Property 1","Property 1","Property 1","Property 2","Property 2","Property 2"),
                      price = c(10000, 12500, 18000, 245000, 512000, 1000000))

> blogDT
         name   price
1: Property 1   10000
2: Property 1   12500
3: Property 1   18000
4: Property 2  245000
5: Property 2  512000
6: Property 2 1000000
----

We want to group by the 'name' column and then have the price on row 1 show on row 2, the price on row 2 on row 3, the price on row 4 on row 5 and the price on row 5 on row 6. To do that we'll introduce a 'lag.price' column:

[source,r]
----

> blogDT[, lag.price := c(NA, price[-.N]), by = name]

> blogDT
         name   price lag.price
1: Property 1   10000        NA
2: Property 1   12500     10000
3: Property 1   18000     12500
4: Property 2  245000        NA
5: Property 2  512000    245000
6: Property 2 1000000    512000
----

Next let's calculate the difference between the two prices:

[source,r]
----

> blogDT[, diff := price - lag.price]
> blogDT
         name   price lag.price   diff
1: Property 1   10000        NA     NA
2: Property 1   12500     10000   2500
3: Property 1   18000     12500   5500
4: Property 2  245000        NA     NA
5: Property 2  512000    245000 267000
6: Property 2 1000000    512000 488000
----

Finally let's order the data table by the biggest price gains:

[source,r]
----

> blogDT[order(-diff)]
         name   price lag.price   diff
1: Property 2 1000000    512000 488000
2: Property 2  512000    245000 267000
3: Property 1   18000     12500   5500
4: Property 1   12500     10000   2500
5: Property 1   10000        NA     NA
6: Property 2  245000        NA     NA
----
SparkR: Error in invokeJava(isStatic = TRUE, className, methodName, ...) :  java.lang.ClassNotFoundException: Failed to load class for data source: csv.
Date: 2015-09-21 22:06:44

I've been wanting to play around with https://spark.apache.org/docs/latest/sparkr.html[SparkR] for a while and over the weekend deciding to explore a large link:[Land Registry CSV file] containing all the sales of properties in the UK over the last 20 years.

First I started up the SparkR shell with the CSV package loaded in: ~~~bash ./spark-1.5.0-bin-hadoop2.6/bin/sparkR --packages com.databricks:spark-csv_2.11:1.2.0 ~~~

Next I tried to read the CSV file into a Spark data frame by modifying one of the examples from the tutorial: ~~~bash > sales \<- read.df(sqlContext, "pp-complete.csv", "csv") 15/09/20 19:13:02 ERROR RBackendHandler: loadDF on org.apache.spark.sql.api.r.SQLUtils failed Error in invokeJava(isStatic = TRUE, className, methodName, \...) : java.lang.ClassNotFoundException: Failed to load class for data source: csv. at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:67) at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:87) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114) at org.apache.spark.sql.api.r.SQLUtils$.loadDF(SQLUtils.scala:156) at org.apache.spark.sql.api.r.SQLUtils.loadDF(SQLUtils.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:132) at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:79) at org.apache.spark.api.r.RBackendH ~~~

As far as I can tell I have loaded in the CSV data source so I'm not sure why that doesn't work.

However, I came across this https://github.com/databricks/spark-csv/issues/79[github issue] which suggested passing in the full package name as the 3rd argument of 'read.df' rather than just 'csv': ~~~bash > sales \<- read.df(sqlContext, "pp-complete.csv", "com.databricks.spark.csv", header="false") > sales DataFrame[C0:string, C1:string, C2:string, C3:string, C4:string, C5:string, C6:string, C7:string, C8:string, C9:string, C10:string, C11:string, C12:string, C13:string, C14:string] ~~~

And that worked much better! We can now carry on and do some slicing and dicing of the data to see if there are any interesting insights.
IntelliJ 14.1.5: Unable to import maven project
Date: 2015-09-30 05:54:54

After a recent IntelliJ upgrade I've been running into the following error when trying to attach the sources of any library being pulled in via Maven:

____
Unable to import maven project
____

It seems like this is a recent issue in the 14.x series and luckily is http://stackoverflow.com/questions/30569909/unable-to-import-maven-project-in-intellij14[reasonably easy to fix] by https://youtrack.jetbrains.com/issue/IDEA-140208[adding the following flag] to the VM options passed to the Maven importer:

[source,text]
----

-Didea.maven3.use.compat.resolver
----

And this is where you need to add it:

image::{{<siteurl>}}/uploads/2015/09/2015-09-30_00-18-17.png[2015 09 30 00 18 17,500]

Cmd + , \-> Build, Execution, Deployment \-> Build Tools \-> Maven \-> Importing
Python: Extracting Excel spreadsheet into CSV files
Date: 2015-08-19 23:27:42

I've been playing around with the http://data.gov.uk/dataset/road-accidents-safety-data[Road Safety open data set] and the download comes with several CSV files and an excel spreadsheet containing the legend.

There are 45 sheets in total and each of them looks like this:

image::{{<siteurl>}}/uploads/2015/08/2015-08-17_23-33-19.png[2015 08 17 23 33 19]

I wanted to create a CSV file for each sheet so that I can import the data set into Neo4j using the http://neo4j.com/docs/stable/query-load-csv.html[LOAD CSV command].

I came across the http://www.python-excel.org/[Python Excel website] which pointed me at the +++<cite>+++xlrd+++</cite>+++ library since I'm working with a pre 2010 Excel file.

The https://secure.simplistix.co.uk/svn/xlrd/trunk/xlrd/doc/xlrd.html?p=4966#sheet.Cell-class[main documentation is very extensive] but I found the https://github.com/python-excel/xlrd[github example] much easier to follow.

I ended up with the following script which iterates through all but the first two sheets in the spreadsheet - the first two sheets contain instructions rather than data:

[source,python]
----

from xlrd import open_workbook
import csv

wb = open_workbook('Road-Accident-Safety-Data-Guide-1979-2004.xls')

for i in range(2, wb.nsheets):
    sheet = wb.sheet_by_index(i)
    print sheet.name
    with open("data/%s.csv" %(sheet.name.replace(" ","")), "w") as file:
        writer = csv.writer(file, delimiter = ",")
        print sheet, sheet.name, sheet.ncols, sheet.nrows

        header = [cell.value for cell in sheet.row(0)]
        writer.writerow(header)

        for row_idx in range(1, sheet.nrows):
            row = [int(cell.value) if isinstance(cell.value, float) else cell.value
                   for cell in sheet.row(row_idx)]
            writer.writerow(row)
----

I've replaced spaces in the sheet name so that the file name on a disk is a bit easier to work with. For some reason the numeric values were all floats whereas I wanted them as ints so I had to explicitly apply that transformation.

Here are a few examples of what the CSV files look like:

[source,bash]
----

$ cat data/1stPointofImpact.csv
code,label
0,Did not impact
1,Front
2,Back
3,Offside
4,Nearside
-1,Data missing or out of range

$ cat data/RoadType.csv
code,label
1,Roundabout
2,One way street
3,Dual carriageway
6,Single carriageway
7,Slip road
9,Unknown
12,One way street/Slip road
-1,Data missing or out of range

$ cat data/Weather.csv
code,label
1,Fine no high winds
2,Raining no high winds
3,Snowing no high winds
4,Fine + high winds
5,Raining + high winds
6,Snowing + high winds
7,Fog or mist
8,Other
9,Unknown
-1,Data missing or out of range
----

And that's it. Not too difficult!
Unix: Stripping first n bytes in a file / Byte Order Mark (BOM)
Date: 2015-08-19 23:27:28

I've previously written http://www.markhneedham.com/blog/2012/10/07/mac-os-x-removing-byte-order-mark-with-an-editor/[a couple] http://www.markhneedham.com/blog/2012/09/03/a-rogue-357273277-utf-8-byte-order-mark/[of blog posts] showing how to strip out the byte order mark (BOM) from CSV files to make loading them into Neo4j easier and today I came across another way to clean up the file using tail.

The BOM is 3 bytes long at the beginning of the file so if we know that a file contains it then we can strip out those first 3 bytes tail like this:

[source,bash]
----

$ time tail -c +4 Casualty7904.csv > Casualty7904_stripped.csv

real	0m31.945s
user	0m31.370s
sys	0m0.518s
----

The -c command is described thus;

[source,bash]
----

-c number
             The location is number bytes.
----

So in this case we start reading at byte 4 (i.e. skipping the first 3 bytes) and then direct the output into a new file.

Although using tail is quite simple, it took 30 seconds to process a 300MB CSV file which might actually be slower than opening the file with a Hex editor and manually deleting the bytes!